Fri Oct 16 10:54:29 PDT 2015 Starting regionserver on c2021.halxg.cloudera.com core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 385910 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 32768 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 32768 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 2015-10-16 10:54:30,881 INFO [main] util.VersionInfo: HBase 2.0.0-SNAPSHOT 2015-10-16 10:54:30,881 INFO [main] util.VersionInfo: Source code repository git://c2020.halxg.cloudera.com/home/stack/hbase.git revision=1458798eb593358fe5415596b2958f2f7e451ea5 2015-10-16 10:54:30,881 INFO [main] util.VersionInfo: Compiled by stack on Tue Oct 13 16:03:56 PDT 2015 2015-10-16 10:54:30,881 INFO [main] util.VersionInfo: From source with checksum 8b3d8d1cdf4b1a671bcbc0304ad1781c 2015-10-16 10:54:31,389 INFO [main] util.ServerCommandLine: env:HBASE_LOGFILE=hbase-stack-regionserver-c2021.halxg.cloudera.com.log 2015-10-16 10:54:31,389 INFO [main] util.ServerCommandLine: env:PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin 2015-10-16 10:54:31,389 INFO [main] util.ServerCommandLine: env:HBASE_REGIONSERVER_OPTS= -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=10102 2015-10-16 10:54:31,389 INFO [main] util.ServerCommandLine: env:JAVA_HOME=/home/stack/bin/jdk/ 2015-10-16 10:54:31,389 INFO [main] util.ServerCommandLine: env:XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt 2015-10-16 10:54:31,389 INFO [main] util.ServerCommandLine: env:LANG=en_US.UTF-8 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:G_BROKEN_FILENAMES=1 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:FACTER_macaddress=e8:39:35:02:86:10 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:MAIL=/var/mail/stack 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:LD_LIBRARY_PATH=:/home/stack/hadoop-2.7.0-SNAPSHOT/lib/native 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:LOGNAME=stack 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:HBASE_REST_OPTS= 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:PWD=/home/stack/hbase 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:HBASE_ROOT_LOGGER=INFO,RFA 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:LESSOPEN=|/usr/bin/lesspipe.sh %s 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:SHELL=/bin/bash 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:HBASE_HEAPSIZE=16000 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:HBASE_ENV_INIT=true 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:QTINC=/usr/lib64/qt-3.3/include 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:CVS_RSH=ssh 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:HBASE_IDENT_STRING=stack 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:HBASE_ZNODE_FILE=/tmp/hbase-stack-regionserver.znode 2015-10-16 10:54:31,390 INFO [main] util.ServerCommandLine: env:SSH_CLIENT=10.20.84.26 49159 22 2015-10-16 10:54:31,391 INFO [main] util.ServerCommandLine: env:HBASE_LOG_PREFIX=hbase-stack-regionserver-c2021.halxg.cloudera.com 2015-10-16 10:54:31,391 INFO [main] util.ServerCommandLine: env:HBASE_LOG_DIR=/home/stack/logs 2015-10-16 10:54:31,391 INFO [main] util.ServerCommandLine: env:USER=stack 2015-10-16 10:54:31,391 INFO [main] util.ServerCommandLine: env:CLASSPATH=/home/stack/conf_hbase:/home/stack/bin/jdk//lib/tools.jar:/home/stack/hbase/bin/..:/home/stack/hbase/bin/../lib/activation-1.1.jar:/home/stack/hbase/bin/../lib/aopalliance-1.0.jar:/home/stack/hbase/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/home/stack/hbase/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/stack/hbase/bin/../lib/api-asn1-api-1.0.0-M20.jar:/home/stack/hbase/bin/../lib/api-util-1.0.0-M20.jar:/home/stack/hbase/bin/../lib/asm-3.1.jar:/home/stack/hbase/bin/../lib/avro-1.7.4.jar:/home/stack/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/home/stack/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/stack/hbase/bin/../lib/commons-cli-1.2.jar:/home/stack/hbase/bin/../lib/commons-codec-1.9.jar:/home/stack/hbase/bin/../lib/commons-collections-3.2.1.jar:/home/stack/hbase/bin/../lib/commons-compress-1.4.1.jar:/home/stack/hbase/bin/../lib/commons-configuration-1.6.jar:/home/stack/hbase/bin/../lib/commons-daemon-1.0.13.jar:/home/stack/hbase/bin/../lib/commons-digester-1.8.jar:/home/stack/hbase/bin/../lib/commons-el-1.0.jar:/home/stack/hbase/bin/../lib/commons-httpclient-3.1.jar:/home/stack/hbase/bin/../lib/commons-io-2.4.jar:/home/stack/hbase/bin/../lib/commons-lang-2.6.jar:/home/stack/hbase/bin/../lib/commons-logging-1.2.jar:/home/stack/hbase/bin/../lib/commons-math-2.2.jar:/home/stack/hbase/bin/../lib/commons-math3-3.1.1.jar:/home/stack/hbase/bin/../lib/commons-net-3.1.jar:/home/stack/hbase/bin/../lib/curator-client-2.7.1.jar:/home/stack/hbase/bin/../lib/curator-framework-2.7.1.jar:/home/stack/hbase/bin/../lib/curator-recipes-2.7.1.jar:/home/stack/hbase/bin/../lib/disruptor-3.3.0.jar:/home/stack/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/home/stack/hbase/bin/../lib/gson-2.2.4.jar:/home/stack/hbase/bin/../lib/guava-12.0.1.jar:/home/stack/hbase/bin/../lib/guice-3.0.jar:/home/stack/hbase/bin/../lib/guice-servlet-3.0.jar:/home/stack/hbase/bin/../lib/hadoop-annotations-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-auth-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-client-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-hdfs-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-app-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-core-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-jobclient-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-shuffle-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-api-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-client-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-server-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hbase-annotations-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-annotations-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-client-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-common-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-common-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-examples-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-external-blockcache-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-hadoop2-compat-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-hadoop-compat-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-it-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-it-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-prefix-tree-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-procedure-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-protocol-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-resource-bundle-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-rest-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-server-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-server-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-shell-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-thrift-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/htrace-core-3.1.0-incubating.jar:/home/stack/hbase/bin/../lib/httpclient-4.2.5.jar:/home/stack/hbase/bin/../lib/httpcore-4.2.4.jar:/home/stack/hbase/bin/../lib/jackson-core-asl-1.9.13.jar:/home/stack/hbase/bin/../lib/jackson-jaxrs-1.9.13.jar:/home/stack/hbase/bin/../lib/jackson-mapper-asl-1.9.13.jar:/home/stack/hbase/bin/../lib/jackson-xc-1.9.13.jar:/home/stack/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/home/stack/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/home/stack/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/home/stack/hbase/bin/../lib/javax.inject-1.jar:/home/stack/hbase/bin/../lib/java-xmlbuilder-0.4.jar:/home/stack/hbase/bin/../lib/jaxb-api-2.2.2.jar:/home/stack/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/stack/hbase/bin/../lib/jcodings-1.0.8.jar:/home/stack/hbase/bin/../lib/jersey-client-1.9.jar:/home/stack/hbase/bin/../lib/jersey-core-1.9.jar:/home/stack/hbase/bin/../lib/jersey-guice-1.9.jar:/home/stack/hbase/bin/../lib/jersey-json-1.9.jar:/home/stack/hbase/bin/../lib/jersey-server-1.9.jar:/home/stack/hbase/bin/../lib/jets3t-0.9.0.jar:/home/stack/hbase/bin/../lib/jettison-1.3.3.jar:/home/stack/hbase/bin/../lib/jetty-6.1.26.jar:/home/stack/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/home/stack/hbase/bin/../lib/jetty-util-6.1.26.jar:/home/stack/hbase/bin/../lib/joni-2.1.2.jar:/home/stack/hbase/bin/../lib/jruby-complete-1.6.8.jar:/home/stack/hbase/bin/../lib/jsch-0.1.42.jar:/home/stack/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/home/stack/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/stack/hbase/bin/../lib/jsr305-1.3.9.jar:/home/stack/hbase/bin/../lib/junit-4.12.jar:/home/stack/hbase/bin/../lib/leveldbjni-all-1.8.jar:/home/stack/hbase/bin/../lib/libthrift-0.9.2.jar:/home/stack/hbase/bin/../lib/log4j-1.2.17.jar:/home/stack/hbase/bin/../lib/metrics-core-2.2.0.jar:/home/stack/hbase/bin/../lib/netty-all-4.0.30.Final.jar:/home/stack/hbase/bin/../lib/paranamer-2.3.jar:/home/stack/hbase/bin/../lib/protobuf-java-2.5.0.jar:/home/stack/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/stack/hbase/bin/../lib/servlet-api-2.5.jar:/home/stack/hbase/bin/../lib/slf4j-api-1.7.7.jar:/home/stack/hbase/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/stack/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/home/stack/hbase/bin/../lib/spymemcached-2.11.6.jar:/home/stack/hbase/bin/../lib/xercesImpl-2.9.1.jar:/home/stack/hbase/bin/../lib/xml-apis-1.3.04.jar:/home/stack/hbase/bin/../lib/xmlenc-0.52.jar:/home/stack/hbase/bin/../lib/xz-1.0.jar:/home/stack/hbase/bin/../lib/zookeeper-3.4.6.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/etc/hadoop:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/*:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/*:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/*:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/*:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/*:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/*:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/*:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/*:/home/stack/hadoop/contrib/capacity-scheduler/*.jar 2015-10-16 10:54:31,391 INFO [main] util.ServerCommandLine: env:HBASE_MASTER_OPTS= -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=10101 2015-10-16 10:54:31,392 INFO [main] util.ServerCommandLine: env:SERVER_GC_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps 2015-10-16 10:54:31,392 INFO [main] util.ServerCommandLine: env:SSH_CONNECTION=10.20.84.26 49159 10.20.84.27 22 2015-10-16 10:54:31,392 INFO [main] util.ServerCommandLine: env:HBASE_JMX_BASE=-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false 2015-10-16 10:54:31,392 INFO [main] util.ServerCommandLine: env:QTDIR=/usr/lib64/qt-3.3 2015-10-16 10:54:31,392 INFO [main] util.ServerCommandLine: env:NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat 2015-10-16 10:54:31,392 INFO [main] util.ServerCommandLine: env:HADOOP_HOME=/home/stack/hadoop 2015-10-16 10:54:31,392 INFO [main] util.ServerCommandLine: env:HBASE_NICENESS=0 2015-10-16 10:54:31,392 INFO [main] util.ServerCommandLine: env:HBASE_OPTS= -XX:+UseConcMarkSweepGC -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:MaxDirectMemorySize=16g -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=10102 -Dhbase.log.dir=/home/stack/logs -Dhbase.log.file=hbase-stack-regionserver-c2021.halxg.cloudera.com.log -Dhbase.home.dir=/home/stack/hbase/bin/.. -Dhbase.id.str=stack -Dhbase.root.logger=INFO,RFA -Djava.library.path=/home/stack/hadoop-2.7.0-SNAPSHOT/lib/native -Dhbase.security.logger=INFO,RFAS 2015-10-16 10:54:31,392 INFO [main] util.ServerCommandLine: env:HBASE_START_FILE=/tmp/hbase-stack-regionserver.autorestart 2015-10-16 10:54:31,392 INFO [main] util.ServerCommandLine: env:HBASE_SECURITY_LOGGER=INFO,RFAS 2015-10-16 10:54:31,392 INFO [main] util.ServerCommandLine: env:HBASE_THRIFT_OPTS= 2015-10-16 10:54:31,392 INFO [main] util.ServerCommandLine: env:HBASE_HOME=/home/stack/hbase/bin/.. 2015-10-16 10:54:31,392 INFO [main] util.ServerCommandLine: env:QTLIB=/usr/lib64/qt-3.3/lib 2015-10-16 10:54:31,392 INFO [main] util.ServerCommandLine: env:SHLVL=3 2015-10-16 10:54:31,393 INFO [main] util.ServerCommandLine: env:HOME=/home/stack 2015-10-16 10:54:31,393 INFO [main] util.ServerCommandLine: env:MALLOC_ARENA_MAX=4 2015-10-16 10:54:31,393 INFO [main] util.ServerCommandLine: vmName=Java HotSpot(TM) 64-Bit Server VM, vmVendor=Oracle Corporation, vmVersion=25.60-b23 2015-10-16 10:54:31,393 INFO [main] util.ServerCommandLine: vmInputArguments=[-Dproc_regionserver, -XX:OnOutOfMemoryError=kill -9 %p, -Xmx16000m, -XX:+UseConcMarkSweepGC, -XX:+UnlockCommercialFeatures, -XX:+FlightRecorder, -XX:MaxDirectMemorySize=16g, -verbose:gc, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -Dcom.sun.management.jmxremote.ssl=false, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.port=10102, -Dhbase.log.dir=/home/stack/logs, -Dhbase.log.file=hbase-stack-regionserver-c2021.halxg.cloudera.com.log, -Dhbase.home.dir=/home/stack/hbase/bin/.., -Dhbase.id.str=stack, -Dhbase.root.logger=INFO,RFA, -Djava.library.path=/home/stack/hadoop-2.7.0-SNAPSHOT/lib/native, -Dhbase.security.logger=INFO,RFAS] 2015-10-16 10:54:31,805 INFO [main] regionserver.RSRpcServices: regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020 server-side HConnection retries=350 2015-10-16 10:54:31,966 INFO [main] ipc.SimpleRpcScheduler: Using deadline as user call queue, count=3 2015-10-16 10:54:31,986 INFO [main] ipc.RpcServer: regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020: started 10 reader(s) listening on port=16020 2015-10-16 10:54:32,054 INFO [main] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2015-10-16 10:54:32,076 INFO [main] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2015-10-16 10:54:32,077 INFO [main] impl.MetricsSystemImpl: HBase metrics system started 2015-10-16 10:54:32,113 INFO [main] hfile.CacheConfig: Allocating LruBlockCache size=1.55 GB, blockSize=64 KB 2015-10-16 10:54:32,127 DEBUG [main] hfile.CacheConfig: Trying to use Internal l2 cache 2015-10-16 10:54:32,138 INFO [main] util.ByteBufferArray: Allocating buffers total=8.00 GB, sizePerBuffer=4 MB, count=2049, direct=true 2015-10-16 10:54:37,351 INFO [main] bucket.BucketCache: Started bucket cache; ioengine=offheap, capacity=8.00 GB, blockSize=64 KB, writerThreadNum=3, writerQLen=64, persistencePath=null, bucketAllocator=org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocator 2015-10-16 10:54:37,353 INFO [main] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 10:54:37,354 INFO [main] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 10:54:37,357 INFO [main] mob.MobFileCache: MobFileCache is initialized, and the cache size is 1000 2015-10-16 10:54:37,969 INFO [main] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-10-16 10:54:38,055 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=regionserver:16020 connecting to ZooKeeper ensemble=c2020.halxg.cloudera.com:2181 2015-10-16 10:54:38,063 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2015-10-16 10:54:38,064 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=c2021.halxg.cloudera.com 2015-10-16 10:54:38,064 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_60 2015-10-16 10:54:38,064 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2015-10-16 10:54:38,064 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/home/stack/bin/jdk1.8.0_60/jre 2015-10-16 10:54:38,064 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/home/stack/conf_hbase:/home/stack/bin/jdk//lib/tools.jar:/home/stack/hbase/bin/..:/home/stack/hbase/bin/../lib/activation-1.1.jar:/home/stack/hbase/bin/../lib/aopalliance-1.0.jar:/home/stack/hbase/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/home/stack/hbase/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/stack/hbase/bin/../lib/api-asn1-api-1.0.0-M20.jar:/home/stack/hbase/bin/../lib/api-util-1.0.0-M20.jar:/home/stack/hbase/bin/../lib/asm-3.1.jar:/home/stack/hbase/bin/../lib/avro-1.7.4.jar:/home/stack/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/home/stack/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/stack/hbase/bin/../lib/commons-cli-1.2.jar:/home/stack/hbase/bin/../lib/commons-codec-1.9.jar:/home/stack/hbase/bin/../lib/commons-collections-3.2.1.jar:/home/stack/hbase/bin/../lib/commons-compress-1.4.1.jar:/home/stack/hbase/bin/../lib/commons-configuration-1.6.jar:/home/stack/hbase/bin/../lib/commons-daemon-1.0.13.jar:/home/stack/hbase/bin/../lib/commons-digester-1.8.jar:/home/stack/hbase/bin/../lib/commons-el-1.0.jar:/home/stack/hbase/bin/../lib/commons-httpclient-3.1.jar:/home/stack/hbase/bin/../lib/commons-io-2.4.jar:/home/stack/hbase/bin/../lib/commons-lang-2.6.jar:/home/stack/hbase/bin/../lib/commons-logging-1.2.jar:/home/stack/hbase/bin/../lib/commons-math-2.2.jar:/home/stack/hbase/bin/../lib/commons-math3-3.1.1.jar:/home/stack/hbase/bin/../lib/commons-net-3.1.jar:/home/stack/hbase/bin/../lib/curator-client-2.7.1.jar:/home/stack/hbase/bin/../lib/curator-framework-2.7.1.jar:/home/stack/hbase/bin/../lib/curator-recipes-2.7.1.jar:/home/stack/hbase/bin/../lib/disruptor-3.3.0.jar:/home/stack/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/home/stack/hbase/bin/../lib/gson-2.2.4.jar:/home/stack/hbase/bin/../lib/guava-12.0.1.jar:/home/stack/hbase/bin/../lib/guice-3.0.jar:/home/stack/hbase/bin/../lib/guice-servlet-3.0.jar:/home/stack/hbase/bin/../lib/hadoop-annotations-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-auth-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-client-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-hdfs-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-app-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-core-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-jobclient-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-shuffle-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-api-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-client-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-server-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hbase-annotations-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-annotations-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-client-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-common-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-common-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-examples-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-external-blockcache-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-hadoop2-compat-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-hadoop-compat-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-it-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-it-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-prefix-tree-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-procedure-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-protocol-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-resource-bundle-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-rest-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-server-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-server-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-shell-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-thrift-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/htrace-core-3.1.0-incubating.jar:/home/stack/hbase/bin/../lib/httpclient-4.2.5.jar:/home/stack/hbase/bin/../lib/httpcore-4.2.4.jar:/home/stack/hbase/bin/../lib/jackson-core-asl-1.9.13.jar:/home/stack/hbase/bin/../lib/jackson-jaxrs-1.9.13.jar:/home/stack/hbase/bin/../lib/jackson-mapper-asl-1.9.13.jar:/home/stack/hbase/bin/../lib/jackson-xc-1.9.13.jar:/home/stack/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/home/stack/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/home/stack/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/home/stack/hbase/bin/../lib/javax.inject-1.jar:/home/stack/hbase/bin/../lib/java-xmlbuilder-0.4.jar:/home/stack/hbase/bin/../lib/jaxb-api-2.2.2.jar:/home/stack/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/stack/hbase/bin/../lib/jcodings-1.0.8.jar:/home/stack/hbase/bin/../lib/jersey-client-1.9.jar:/home/stack/hbase/bin/../lib/jersey-core-1.9.jar:/home/stack/hbase/bin/../lib/jersey-guice-1.9.jar:/home/stack/hbase/bin/../lib/jersey-json-1.9.jar:/home/stack/hbase/bin/../lib/jersey-server-1.9.jar:/home/stack/hbase/bin/../lib/jets3t-0.9.0.jar:/home/stack/hbase/bin/../lib/jettison-1.3.3.jar:/home/stack/hbase/bin/../lib/jetty-6.1.26.jar:/home/stack/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/home/stack/hbase/bin/../lib/jetty-util-6.1.26.jar:/home/stack/hbase/bin/../lib/joni-2.1.2.jar:/home/stack/hbase/bin/../lib/jruby-complete-1.6.8.jar:/home/stack/hbase/bin/../lib/jsch-0.1.42.jar:/home/stack/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/home/stack/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/stack/hbase/bin/../lib/jsr305-1.3.9.jar:/home/stack/hbase/bin/../lib/junit-4.12.jar:/home/stack/hbase/bin/../lib/leveldbjni-all-1.8.jar:/home/stack/hbase/bin/../lib/libthrift-0.9.2.jar:/home/stack/hbase/bin/../lib/log4j-1.2.17.jar:/home/stack/hbase/bin/../lib/metrics-core-2.2.0.jar:/home/stack/hbase/bin/../lib/netty-all-4.0.30.Final.jar:/home/stack/hbase/bin/../lib/paranamer-2.3.jar:/home/stack/hbase/bin/../lib/protobuf-java-2.5.0.jar:/home/stack/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/stack/hbase/bin/../lib/servlet-api-2.5.jar:/home/stack/hbase/bin/../lib/slf4j-api-1.7.7.jar:/home/stack/hbase/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/stack/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/home/stack/hbase/bin/../lib/spymemcached-2.11.6.jar:/home/stack/hbase/bin/../lib/xercesImpl-2.9.1.jar:/home/stack/hbase/bin/../lib/xml-apis-1.3.04.jar:/home/stack/hbase/bin/../lib/xmlenc-0.52.jar:/home/stack/hbase/bin/../lib/xz-1.0.jar:/home/stack/hbase/bin/../lib/zookeeper-3.4.6.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/etc/hadoop:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/servlet-api-2.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/log4j-1.2.17.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-codec-1.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jettison-1.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jersey-server-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/curator-client-2.6.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/activation-1.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jersey-json-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-lang-2.6.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jetty-6.1.26.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-net-3.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/junit-4.11.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/paranamer-2.3.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-digester-1.8.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/htrace-core-3.0.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/hadoop-annotations-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jsch-0.1.42.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/gson-2.2.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/curator-framework-2.6.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/hadoop-auth-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jsp-api-2.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/avro-1.7.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jersey-core-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-io-2.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/xz-1.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-cli-1.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/xmlenc-0.52.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/asm-3.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/guava-11.0.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/hadoop-nfs-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/hadoop-common-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/hadoop-common-2.7.0-SNAPSHOT-tests.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/asm-3.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/hadoop-hdfs-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/hadoop-hdfs-2.7.0-SNAPSHOT-tests.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jettison-1.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/guice-3.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/activation-1.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/javax.inject-1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jline-0.9.94.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/xz-1.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/asm-3.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-common-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-registry-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-client-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-api-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-common-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.0-SNAPSHOT-tests.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.0-SNAPSHOT.jar:/home/stack/hadoop/contrib/capacity-scheduler/*.jar 2015-10-16 10:54:38,065 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/home/stack/hadoop-2.7.0-SNAPSHOT/lib/native 2015-10-16 10:54:38,065 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2015-10-16 10:54:38,065 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler= 2015-10-16 10:54:38,065 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2015-10-16 10:54:38,065 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2015-10-16 10:54:38,065 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-358.23.2.el6.x86_64 2015-10-16 10:54:38,065 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=stack 2015-10-16 10:54:38,065 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/stack 2015-10-16 10:54:38,065 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/stack/hbase-2.0.0-SNAPSHOT 2015-10-16 10:54:38,066 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=c2020.halxg.cloudera.com:2181 sessionTimeout=90000 watcher=regionserver:160200x0, quorum=c2020.halxg.cloudera.com:2181, baseZNode=/hbase 2015-10-16 10:54:38,084 INFO [main-SendThread(c2020.halxg.cloudera.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server c2020.halxg.cloudera.com/10.20.84.26:2181. Will not attempt to authenticate using SASL (unknown error) 2015-10-16 10:54:38,091 INFO [main-SendThread(c2020.halxg.cloudera.com:2181)] zookeeper.ClientCnxn: Socket connection established to c2020.halxg.cloudera.com/10.20.84.26:2181, initiating session 2015-10-16 10:54:38,110 INFO [main-SendThread(c2020.halxg.cloudera.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server c2020.halxg.cloudera.com/10.20.84.26:2181, sessionid = 0x15071c9e6740003, negotiated timeout = 90000 2015-10-16 10:54:38,124 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: starting 2015-10-16 10:54:38,124 INFO [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: starting 2015-10-16 10:54:38,125 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=0 queue=0 2015-10-16 10:54:38,125 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=1 queue=1 2015-10-16 10:54:38,125 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=2 queue=2 2015-10-16 10:54:38,125 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=3 queue=0 2015-10-16 10:54:38,126 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=4 queue=1 2015-10-16 10:54:38,126 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=5 queue=2 2015-10-16 10:54:38,126 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=6 queue=0 2015-10-16 10:54:38,126 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=7 queue=1 2015-10-16 10:54:38,127 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=8 queue=2 2015-10-16 10:54:38,127 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=9 queue=0 2015-10-16 10:54:38,127 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=10 queue=1 2015-10-16 10:54:38,127 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=11 queue=2 2015-10-16 10:54:38,128 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=12 queue=0 2015-10-16 10:54:38,128 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=13 queue=1 2015-10-16 10:54:38,128 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=14 queue=2 2015-10-16 10:54:38,128 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=15 queue=0 2015-10-16 10:54:38,128 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=16 queue=1 2015-10-16 10:54:38,129 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=17 queue=2 2015-10-16 10:54:38,129 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=18 queue=0 2015-10-16 10:54:38,129 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=19 queue=1 2015-10-16 10:54:38,129 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=20 queue=2 2015-10-16 10:54:38,129 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=21 queue=0 2015-10-16 10:54:38,129 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=22 queue=1 2015-10-16 10:54:38,130 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=23 queue=2 2015-10-16 10:54:38,130 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=24 queue=0 2015-10-16 10:54:38,130 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=25 queue=1 2015-10-16 10:54:38,130 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=26 queue=2 2015-10-16 10:54:38,130 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=27 queue=0 2015-10-16 10:54:38,131 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=28 queue=1 2015-10-16 10:54:38,131 DEBUG [main] ipc.RpcExecutor: B.default Start Handler index=29 queue=2 2015-10-16 10:54:38,131 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=0 queue=0 2015-10-16 10:54:38,131 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=1 queue=1 2015-10-16 10:54:38,132 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=2 queue=0 2015-10-16 10:54:38,132 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=3 queue=1 2015-10-16 10:54:38,132 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=4 queue=0 2015-10-16 10:54:38,132 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=5 queue=1 2015-10-16 10:54:38,132 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=6 queue=0 2015-10-16 10:54:38,133 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=7 queue=1 2015-10-16 10:54:38,133 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=8 queue=0 2015-10-16 10:54:38,133 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=9 queue=1 2015-10-16 10:54:38,133 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=10 queue=0 2015-10-16 10:54:38,133 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=11 queue=1 2015-10-16 10:54:38,134 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=12 queue=0 2015-10-16 10:54:38,134 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=13 queue=1 2015-10-16 10:54:38,134 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=14 queue=0 2015-10-16 10:54:38,134 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=15 queue=1 2015-10-16 10:54:38,135 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=16 queue=0 2015-10-16 10:54:38,135 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=17 queue=1 2015-10-16 10:54:38,135 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=18 queue=0 2015-10-16 10:54:38,135 DEBUG [main] ipc.RpcExecutor: Priority Start Handler index=19 queue=1 2015-10-16 10:54:38,135 DEBUG [main] ipc.RpcExecutor: Replication Start Handler index=0 queue=0 2015-10-16 10:54:38,136 DEBUG [main] ipc.RpcExecutor: Replication Start Handler index=1 queue=0 2015-10-16 10:54:38,136 DEBUG [main] ipc.RpcExecutor: Replication Start Handler index=2 queue=0 2015-10-16 10:54:38,206 INFO [main] mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2015-10-16 10:54:38,211 INFO [main] http.HttpRequestLog: Http request log for http.requests.regionserver is not defined 2015-10-16 10:54:38,224 INFO [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2015-10-16 10:54:38,224 INFO [main] http.HttpServer: Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) 2015-10-16 10:54:38,227 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver 2015-10-16 10:54:38,227 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2015-10-16 10:54:38,227 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2015-10-16 10:54:38,243 INFO [main] http.HttpServer: Jetty bound to port 16030 2015-10-16 10:54:38,243 INFO [main] mortbay.log: jetty-6.1.26 2015-10-16 10:54:38,591 INFO [main] mortbay.log: Started SelectChannelConnector@0.0.0.0:16030 2015-10-16 10:54:38,630 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2ebc2425 connecting to ZooKeeper ensemble=c2020.halxg.cloudera.com:2181 2015-10-16 10:54:38,631 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] zookeeper.ZooKeeper: Initiating client connection, connectString=c2020.halxg.cloudera.com:2181 sessionTimeout=90000 watcher=hconnection-0x2ebc24250x0, quorum=c2020.halxg.cloudera.com:2181, baseZNode=/hbase 2015-10-16 10:54:38,633 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-SendThread(c2020.halxg.cloudera.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server c2020.halxg.cloudera.com/10.20.84.26:2181. Will not attempt to authenticate using SASL (unknown error) 2015-10-16 10:54:38,636 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-SendThread(c2020.halxg.cloudera.com:2181)] zookeeper.ClientCnxn: Socket connection established to c2020.halxg.cloudera.com/10.20.84.26:2181, initiating session 2015-10-16 10:54:38,641 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-SendThread(c2020.halxg.cloudera.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server c2020.halxg.cloudera.com/10.20.84.26:2181, sessionid = 0x15071c9e6740004, negotiated timeout = 90000 2015-10-16 10:54:38,741 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] ipc.AbstractRpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15b84315, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null 2015-10-16 10:54:38,741 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] ipc.AsyncRpcClient: Starting async Hbase RPC client 2015-10-16 10:54:38,741 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] ipc.AsyncRpcClient: Create NioEventLoopGroup with maxThreads = 0 2015-10-16 10:54:38,776 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] ipc.AsyncRpcClient: Create global event loop group NioEventLoopGroup 2015-10-16 10:54:38,776 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] ipc.AsyncRpcClient: Use global event loop group NioEventLoopGroup 2015-10-16 10:54:38,788 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.HRegionServer: ClusterId : d21d8426-2daa-4054-857d-7111b350a3b4 2015-10-16 10:54:38,793 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] procedure.RegionServerProcedureManagerHost: Procedure flush-table-proc is initializing 2015-10-16 10:54:38,812 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] zookeeper.RecoverableZooKeeper: Node /hbase/flush-table-proc/acquired already exists 2015-10-16 10:54:38,819 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] procedure.RegionServerProcedureManagerHost: Procedure flush-table-proc is initialized 2015-10-16 10:54:38,819 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] procedure.RegionServerProcedureManagerHost: Procedure online-snapshot is initializing 2015-10-16 10:54:38,824 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] zookeeper.RecoverableZooKeeper: Node /hbase/online-snapshot/acquired already exists 2015-10-16 10:54:38,826 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] procedure.RegionServerProcedureManagerHost: Procedure online-snapshot is initialized 2015-10-16 10:54:38,831 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.MemStoreFlusher: globalMemStoreLimit=6.2 G, globalMemStoreLimitLowMark=5.9 G, maxHeap=15.5 G 2015-10-16 10:54:38,840 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] compactions.PressureAwareCompactionThroughputController: Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2015-10-16 10:54:38,841 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.HRegionServer: CompactionChecker runs every 10sec 2015-10-16 10:54:38,845 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] ipc.AbstractRpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74da4def, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=c2021.halxg.cloudera.com/10.20.84.27:0 2015-10-16 10:54:38,845 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] ipc.AsyncRpcClient: Starting async Hbase RPC client 2015-10-16 10:54:38,845 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] ipc.AsyncRpcClient: Use global event loop group NioEventLoopGroup 2015-10-16 10:54:38,850 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.ShutdownHook: Installed shutdown hook thread: Shutdownhook:regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020 2015-10-16 10:54:38,876 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.RegionServerCoprocessorHost: System coprocessor loading is enabled 2015-10-16 10:54:38,876 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.RegionServerCoprocessorHost: Table coprocessor loading is enabled 2015-10-16 10:54:38,881 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.HRegionServer: reportForDuty to master=c2020.halxg.cloudera.com,16000,1445018070399 with port=16020, startcode=1445018072095 2015-10-16 10:54:39,117 DEBUG [AsyncRpcChannel-pool2-t1] ipc.AsyncRpcChannel: Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-10-16 10:54:39,313 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.HRegionServer: Config from master: hbase.rootdir=hdfs://c2020.halxg.cloudera.com:8020/hbase 2015-10-16 10:54:39,313 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.HRegionServer: Config from master: fs.defaultFS=hdfs://c2020.halxg.cloudera.com:8020 2015-10-16 10:54:39,313 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.HRegionServer: Config from master: hbase.master.info.port=16010 2015-10-16 10:54:39,329 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 10:54:39,331 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.HRegionServer: logdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095 2015-10-16 10:54:39,557 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.Replication: ReplicationStatisticsThread 300 2015-10-16 10:54:39,580 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] wal.WALFactory: Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-10-16 10:54:39,591 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.MetricsRegionServerWrapperImpl: Computing regionserver metrics every 5000 milliseconds 2015-10-16 10:54:39,598 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] executor.ExecutorService: Starting executor service name=RS_OPEN_REGION-c2021:16020, corePoolSize=3, maxPoolSize=3 2015-10-16 10:54:39,598 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] executor.ExecutorService: Starting executor service name=RS_OPEN_META-c2021:16020, corePoolSize=1, maxPoolSize=1 2015-10-16 10:54:39,598 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] executor.ExecutorService: Starting executor service name=RS_CLOSE_REGION-c2021:16020, corePoolSize=3, maxPoolSize=3 2015-10-16 10:54:39,598 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] executor.ExecutorService: Starting executor service name=RS_CLOSE_META-c2021:16020, corePoolSize=1, maxPoolSize=1 2015-10-16 10:54:39,599 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] executor.ExecutorService: Starting executor service name=RS_LOG_REPLAY_OPS-c2021:16020, corePoolSize=2, maxPoolSize=2 2015-10-16 10:54:39,600 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] executor.ExecutorService: Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-c2021:16020, corePoolSize=3, maxPoolSize=3 2015-10-16 10:54:39,608 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.ReplicationSourceManager: Current list of replicators: [c2020.halxg.cloudera.com,16000,1445018070399, c2021.halxg.cloudera.com,16020,1445018072095] other RSs: [c2020.halxg.cloudera.com,16000,1445017560853, c2020.halxg.cloudera.com,16000,1445018070399, c2021.halxg.cloudera.com,16020,1445017562579, c2021.halxg.cloudera.com,16020,1445018072095] 2015-10-16 10:54:39,675 INFO [SplitLogWorker-c2021:16020] regionserver.SplitLogWorker: SplitLogWorker c2021.halxg.cloudera.com,16020,1445018072095 starting 2015-10-16 10:54:39,677 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.HeapMemoryManager: Starting HeapMemoryTuner chore. 2015-10-16 10:54:39,682 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] regionserver.HRegionServer: Serving as c2021.halxg.cloudera.com,16020,1445018072095, RpcServer on c2021.halxg.cloudera.com/10.20.84.27:16020, sessionid=0x15071c9e6740003 2015-10-16 10:54:39,682 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] procedure.RegionServerProcedureManagerHost: Procedure flush-table-proc is starting 2015-10-16 10:54:39,682 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] flush.RegionServerFlushTableProcedureManager: Start region server flush procedure manager c2021.halxg.cloudera.com,16020,1445018072095 2015-10-16 10:54:39,682 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] procedure.ZKProcedureMemberRpcs: Starting procedure member 'c2021.halxg.cloudera.com,16020,1445018072095' 2015-10-16 10:54:39,682 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] procedure.ZKProcedureMemberRpcs: Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-10-16 10:54:39,683 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] procedure.ZKProcedureMemberRpcs: Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-10-16 10:54:39,684 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] procedure.RegionServerProcedureManagerHost: Procedure flush-table-proc is started 2015-10-16 10:54:39,684 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] procedure.RegionServerProcedureManagerHost: Procedure online-snapshot is starting 2015-10-16 10:54:39,684 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] snapshot.RegionServerSnapshotManager: Start Snapshot Manager c2021.halxg.cloudera.com,16020,1445018072095 2015-10-16 10:54:39,684 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] procedure.ZKProcedureMemberRpcs: Starting procedure member 'c2021.halxg.cloudera.com,16020,1445018072095' 2015-10-16 10:54:39,684 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] procedure.ZKProcedureMemberRpcs: Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-10-16 10:54:39,685 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] procedure.ZKProcedureMemberRpcs: Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-10-16 10:54:39,686 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] procedure.RegionServerProcedureManagerHost: Procedure online-snapshot is started 2015-10-16 10:54:39,713 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] quotas.RegionServerQuotaManager: Quota support disabled 2015-10-16 10:54:42,849 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] wal.FSHLog: WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=c2021.halxg.cloudera.com%2C16020%2C1445018072095, suffix=, logDir=hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095, archiveDir=hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs 2015-10-16 10:54:43,102 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] hdfs.DFSClient: Slow waitForAckedSeqno took 173ms (threshold=20ms) 2015-10-16 10:54:43,116 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] wal.FSHLog: Slow sync cost: 190 ms, current pipeline: [] 2015-10-16 10:54:43,119 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020] wal.FSHLog: New WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445018082849 2015-10-16 10:54:52,033 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:41984; # active connections: 1 2015-10-16 10:54:52,156 INFO [PriorityRpcServer.handler=0,queue=0,port=16020] regionserver.RSRpcServices: Open t,,1445018090873.07b76eb51213346d939cd30591ffe6d2. 2015-10-16 10:54:52,350 DEBUG [RS_OPEN_REGION-c2021:16020-0] regionserver.HRegion: Opening region: {ENCODED => 07b76eb51213346d939cd30591ffe6d2, NAME => 't,,1445018090873.07b76eb51213346d939cd30591ffe6d2.', STARTKEY => '', ENDKEY => ''} 2015-10-16 10:54:52,542 DEBUG [RS_OPEN_REGION-c2021:16020-0] regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl for table t 07b76eb51213346d939cd30591ffe6d2 2015-10-16 10:54:52,542 DEBUG [RS_OPEN_REGION-c2021:16020-0] regionserver.HRegion: Instantiated t,,1445018090873.07b76eb51213346d939cd30591ffe6d2. 2015-10-16 10:54:52,614 INFO [StoreOpener-07b76eb51213346d939cd30591ffe6d2-1] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 10:54:52,626 INFO [StoreOpener-07b76eb51213346d939cd30591ffe6d2-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-10-16 10:54:52,631 DEBUG [StoreOpener-07b76eb51213346d939cd30591ffe6d2-1] regionserver.HRegionFileSystem: No StoreFiles for: hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t/07b76eb51213346d939cd30591ffe6d2/t 2015-10-16 10:54:52,654 DEBUG [RS_OPEN_REGION-c2021:16020-0] regionserver.HRegion: Found 0 recovered edits file(s) under hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t/07b76eb51213346d939cd30591ffe6d2 2015-10-16 10:54:52,665 DEBUG [RS_OPEN_REGION-c2021:16020-0] regionserver.FlushLargeStoresPolicy: hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-10-16 10:54:52,682 DEBUG [RS_OPEN_REGION-c2021:16020-0] wal.WALSplitter: Wrote region seqId=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t/07b76eb51213346d939cd30591ffe6d2/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-10-16 10:54:52,682 INFO [RS_OPEN_REGION-c2021:16020-0] regionserver.HRegion: Onlined 07b76eb51213346d939cd30591ffe6d2; next sequenceid=2 2015-10-16 10:54:52,758 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629666_1099518554296] hdfs.DFSClient: Slow ReadProcessor read fields took 9632ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 828701, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 10:54:52,759 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 10:54:52,760 INFO [PostOpenDeployTasks:07b76eb51213346d939cd30591ffe6d2] regionserver.HRegionServer: Post open deploy tasks for t,,1445018090873.07b76eb51213346d939cd30591ffe6d2. 2015-10-16 10:54:52,786 DEBUG [PostOpenDeployTasks:07b76eb51213346d939cd30591ffe6d2] regionserver.HRegionServer: Finished post open deploy task for t,,1445018090873.07b76eb51213346d939cd30591ffe6d2. 2015-10-16 10:54:52,788 DEBUG [RS_OPEN_REGION-c2021:16020-0] handler.OpenRegionHandler: Opened t,,1445018090873.07b76eb51213346d939cd30591ffe6d2. on c2021.halxg.cloudera.com,16020,1445018072095 2015-10-16 10:55:12,036 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:41989; # active connections: 2 2015-10-16 10:55:12,140 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629666_1099518554296] hdfs.DFSClient: Slow ReadProcessor read fields took 19380ms (threshold=20ms); ack: seqno: 2 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1139614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 10:55:16,637 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629666_1099518554296] hdfs.DFSClient: Slow ReadProcessor read fields took 4496ms (threshold=20ms); ack: seqno: 3 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950642, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 10:55:19,334 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629666_1099518554296] hdfs.DFSClient: Slow ReadProcessor read fields took 2696ms (threshold=20ms); ack: seqno: 4 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 980716, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 10:55:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 10:55:51,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629666_1099518554296] hdfs.DFSClient: Slow ReadProcessor read fields took 1950ms (threshold=20ms); ack: seqno: 5 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873429, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 10:56:00,039 INFO [main-EventThread] replication.ReplicationTrackerZKImpl: /hbase/rs/c2020.halxg.cloudera.com,16000,1445017560853 znode expired, triggering replicatorRemoved event 2015-10-16 10:56:00,048 INFO [main-EventThread] replication.ReplicationTrackerZKImpl: /hbase/rs/c2021.halxg.cloudera.com,16020,1445017562579 znode expired, triggering replicatorRemoved event 2015-10-16 10:56:33,176 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl: Atomically moving c2020.halxg.cloudera.com,16000,1445017560853's WALs to my queue 2015-10-16 10:56:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 10:57:14,166 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl: Atomically moving c2021.halxg.cloudera.com,16020,1445017562579's WALs to my queue 2015-10-16 10:57:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 10:58:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 10:59:14,944 INFO [PriorityRpcServer.handler=17,queue=1,port=16020] regionserver.RSRpcServices: Open t1,10,1445018354106.590f2ef9a43c9fe507f0f1fb415ca50e. 2015-10-16 10:59:14,955 INFO [PriorityRpcServer.handler=17,queue=1,port=16020] regionserver.RSRpcServices: Open t1,30,1445018354106.7f06f03dc15543479e658fb409909ffa. 2015-10-16 10:59:14,957 INFO [PriorityRpcServer.handler=17,queue=1,port=16020] regionserver.RSRpcServices: Open t1,20,1445018354106.6f7ef2f09841078548d8e36d1a79c281. 2015-10-16 10:59:14,957 DEBUG [RS_OPEN_REGION-c2021:16020-1] regionserver.HRegion: Opening region: {ENCODED => 590f2ef9a43c9fe507f0f1fb415ca50e, NAME => 't1,10,1445018354106.590f2ef9a43c9fe507f0f1fb415ca50e.', STARTKEY => '10', ENDKEY => '20'} 2015-10-16 10:59:14,960 INFO [PriorityRpcServer.handler=17,queue=1,port=16020] regionserver.RSRpcServices: Open t1,,1445018354106.38cc05fd57524229b6809366776d6542. 2015-10-16 10:59:14,961 DEBUG [RS_OPEN_REGION-c2021:16020-1] regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl for table t1 590f2ef9a43c9fe507f0f1fb415ca50e 2015-10-16 10:59:14,960 DEBUG [RS_OPEN_REGION-c2021:16020-2] regionserver.HRegion: Opening region: {ENCODED => 7f06f03dc15543479e658fb409909ffa, NAME => 't1,30,1445018354106.7f06f03dc15543479e658fb409909ffa.', STARTKEY => '30', ENDKEY => '40'} 2015-10-16 10:59:14,961 DEBUG [RS_OPEN_REGION-c2021:16020-0] regionserver.HRegion: Opening region: {ENCODED => 6f7ef2f09841078548d8e36d1a79c281, NAME => 't1,20,1445018354106.6f7ef2f09841078548d8e36d1a79c281.', STARTKEY => '20', ENDKEY => '30'} 2015-10-16 10:59:14,961 DEBUG [RS_OPEN_REGION-c2021:16020-1] regionserver.HRegion: Instantiated t1,10,1445018354106.590f2ef9a43c9fe507f0f1fb415ca50e. 2015-10-16 10:59:14,962 DEBUG [RS_OPEN_REGION-c2021:16020-2] regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl for table t1 7f06f03dc15543479e658fb409909ffa 2015-10-16 10:59:14,962 DEBUG [RS_OPEN_REGION-c2021:16020-2] regionserver.HRegion: Instantiated t1,30,1445018354106.7f06f03dc15543479e658fb409909ffa. 2015-10-16 10:59:14,962 INFO [PriorityRpcServer.handler=17,queue=1,port=16020] regionserver.RSRpcServices: Open t1,40,1445018354106.b982e41c556da4de25932dd66dee6ea7. 2015-10-16 10:59:14,962 DEBUG [RS_OPEN_REGION-c2021:16020-0] regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl for table t1 6f7ef2f09841078548d8e36d1a79c281 2015-10-16 10:59:14,963 DEBUG [RS_OPEN_REGION-c2021:16020-0] regionserver.HRegion: Instantiated t1,20,1445018354106.6f7ef2f09841078548d8e36d1a79c281. 2015-10-16 10:59:14,976 INFO [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 10:59:14,977 INFO [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-10-16 10:59:14,980 DEBUG [StoreOpener-590f2ef9a43c9fe507f0f1fb415ca50e-1] regionserver.HRegionFileSystem: No StoreFiles for: hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t1/590f2ef9a43c9fe507f0f1fb415ca50e/f1 2015-10-16 10:59:14,982 DEBUG [RS_OPEN_REGION-c2021:16020-1] regionserver.HRegion: Found 0 recovered edits file(s) under hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t1/590f2ef9a43c9fe507f0f1fb415ca50e 2015-10-16 10:59:14,985 INFO [StoreOpener-6f7ef2f09841078548d8e36d1a79c281-1] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 10:59:14,985 INFO [StoreOpener-7f06f03dc15543479e658fb409909ffa-1] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 10:59:14,985 INFO [StoreOpener-6f7ef2f09841078548d8e36d1a79c281-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-10-16 10:59:14,986 DEBUG [RS_OPEN_REGION-c2021:16020-1] regionserver.FlushLargeStoresPolicy: hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-10-16 10:59:14,986 INFO [StoreOpener-7f06f03dc15543479e658fb409909ffa-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-10-16 10:59:14,988 DEBUG [StoreOpener-6f7ef2f09841078548d8e36d1a79c281-1] regionserver.HRegionFileSystem: No StoreFiles for: hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t1/6f7ef2f09841078548d8e36d1a79c281/f1 2015-10-16 10:59:14,988 DEBUG [StoreOpener-7f06f03dc15543479e658fb409909ffa-1] regionserver.HRegionFileSystem: No StoreFiles for: hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t1/7f06f03dc15543479e658fb409909ffa/f1 2015-10-16 10:59:14,989 DEBUG [RS_OPEN_REGION-c2021:16020-0] regionserver.HRegion: Found 0 recovered edits file(s) under hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t1/6f7ef2f09841078548d8e36d1a79c281 2015-10-16 10:59:14,990 DEBUG [RS_OPEN_REGION-c2021:16020-2] regionserver.HRegion: Found 0 recovered edits file(s) under hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t1/7f06f03dc15543479e658fb409909ffa 2015-10-16 10:59:14,992 DEBUG [RS_OPEN_REGION-c2021:16020-0] regionserver.FlushLargeStoresPolicy: hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-10-16 10:59:14,993 DEBUG [RS_OPEN_REGION-c2021:16020-2] regionserver.FlushLargeStoresPolicy: hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-10-16 10:59:15,000 DEBUG [RS_OPEN_REGION-c2021:16020-1] wal.WALSplitter: Wrote region seqId=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t1/590f2ef9a43c9fe507f0f1fb415ca50e/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-10-16 10:59:15,001 INFO [RS_OPEN_REGION-c2021:16020-1] regionserver.HRegion: Onlined 590f2ef9a43c9fe507f0f1fb415ca50e; next sequenceid=2 2015-10-16 10:59:15,004 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629666_1099518554296] hdfs.DFSClient: Slow ReadProcessor read fields took 23719ms (threshold=20ms); ack: seqno: 6 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 856019, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 10:59:15,005 INFO [PostOpenDeployTasks:590f2ef9a43c9fe507f0f1fb415ca50e] regionserver.HRegionServer: Post open deploy tasks for t1,10,1445018354106.590f2ef9a43c9fe507f0f1fb415ca50e. 2015-10-16 10:59:15,011 DEBUG [PostOpenDeployTasks:590f2ef9a43c9fe507f0f1fb415ca50e] regionserver.HRegionServer: Finished post open deploy task for t1,10,1445018354106.590f2ef9a43c9fe507f0f1fb415ca50e. 2015-10-16 10:59:15,011 DEBUG [RS_OPEN_REGION-c2021:16020-1] handler.OpenRegionHandler: Opened t1,10,1445018354106.590f2ef9a43c9fe507f0f1fb415ca50e. on c2021.halxg.cloudera.com,16020,1445018072095 2015-10-16 10:59:15,013 DEBUG [RS_OPEN_REGION-c2021:16020-1] regionserver.HRegion: Opening region: {ENCODED => 38cc05fd57524229b6809366776d6542, NAME => 't1,,1445018354106.38cc05fd57524229b6809366776d6542.', STARTKEY => '', ENDKEY => '10'} 2015-10-16 10:59:15,014 DEBUG [RS_OPEN_REGION-c2021:16020-1] regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl for table t1 38cc05fd57524229b6809366776d6542 2015-10-16 10:59:15,014 DEBUG [RS_OPEN_REGION-c2021:16020-1] regionserver.HRegion: Instantiated t1,,1445018354106.38cc05fd57524229b6809366776d6542. 2015-10-16 10:59:15,017 DEBUG [RS_OPEN_REGION-c2021:16020-2] wal.WALSplitter: Wrote region seqId=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t1/7f06f03dc15543479e658fb409909ffa/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-10-16 10:59:15,017 INFO [RS_OPEN_REGION-c2021:16020-2] regionserver.HRegion: Onlined 7f06f03dc15543479e658fb409909ffa; next sequenceid=2 2015-10-16 10:59:15,021 INFO [PostOpenDeployTasks:7f06f03dc15543479e658fb409909ffa] regionserver.HRegionServer: Post open deploy tasks for t1,30,1445018354106.7f06f03dc15543479e658fb409909ffa. 2015-10-16 10:59:15,025 DEBUG [RS_OPEN_REGION-c2021:16020-0] wal.WALSplitter: Wrote region seqId=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t1/6f7ef2f09841078548d8e36d1a79c281/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-10-16 10:59:15,026 INFO [RS_OPEN_REGION-c2021:16020-0] regionserver.HRegion: Onlined 6f7ef2f09841078548d8e36d1a79c281; next sequenceid=2 2015-10-16 10:59:15,026 DEBUG [PostOpenDeployTasks:7f06f03dc15543479e658fb409909ffa] regionserver.HRegionServer: Finished post open deploy task for t1,30,1445018354106.7f06f03dc15543479e658fb409909ffa. 2015-10-16 10:59:15,027 DEBUG [RS_OPEN_REGION-c2021:16020-2] handler.OpenRegionHandler: Opened t1,30,1445018354106.7f06f03dc15543479e658fb409909ffa. on c2021.halxg.cloudera.com,16020,1445018072095 2015-10-16 10:59:15,028 DEBUG [RS_OPEN_REGION-c2021:16020-2] regionserver.HRegion: Opening region: {ENCODED => b982e41c556da4de25932dd66dee6ea7, NAME => 't1,40,1445018354106.b982e41c556da4de25932dd66dee6ea7.', STARTKEY => '40', ENDKEY => ''} 2015-10-16 10:59:15,029 INFO [PostOpenDeployTasks:6f7ef2f09841078548d8e36d1a79c281] regionserver.HRegionServer: Post open deploy tasks for t1,20,1445018354106.6f7ef2f09841078548d8e36d1a79c281. 2015-10-16 10:59:15,029 DEBUG [RS_OPEN_REGION-c2021:16020-2] regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl for table t1 b982e41c556da4de25932dd66dee6ea7 2015-10-16 10:59:15,029 DEBUG [RS_OPEN_REGION-c2021:16020-2] regionserver.HRegion: Instantiated t1,40,1445018354106.b982e41c556da4de25932dd66dee6ea7. 2015-10-16 10:59:15,034 INFO [StoreOpener-38cc05fd57524229b6809366776d6542-1] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 10:59:15,034 DEBUG [PostOpenDeployTasks:6f7ef2f09841078548d8e36d1a79c281] regionserver.HRegionServer: Finished post open deploy task for t1,20,1445018354106.6f7ef2f09841078548d8e36d1a79c281. 2015-10-16 10:59:15,035 DEBUG [RS_OPEN_REGION-c2021:16020-0] handler.OpenRegionHandler: Opened t1,20,1445018354106.6f7ef2f09841078548d8e36d1a79c281. on c2021.halxg.cloudera.com,16020,1445018072095 2015-10-16 10:59:15,034 INFO [StoreOpener-38cc05fd57524229b6809366776d6542-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-10-16 10:59:15,037 DEBUG [StoreOpener-38cc05fd57524229b6809366776d6542-1] regionserver.HRegionFileSystem: No StoreFiles for: hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t1/38cc05fd57524229b6809366776d6542/f1 2015-10-16 10:59:15,039 DEBUG [RS_OPEN_REGION-c2021:16020-1] regionserver.HRegion: Found 0 recovered edits file(s) under hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t1/38cc05fd57524229b6809366776d6542 2015-10-16 10:59:15,042 INFO [StoreOpener-b982e41c556da4de25932dd66dee6ea7-1] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 10:59:15,043 INFO [StoreOpener-b982e41c556da4de25932dd66dee6ea7-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-10-16 10:59:15,043 DEBUG [RS_OPEN_REGION-c2021:16020-1] regionserver.FlushLargeStoresPolicy: hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-10-16 10:59:15,044 DEBUG [StoreOpener-b982e41c556da4de25932dd66dee6ea7-1] regionserver.HRegionFileSystem: No StoreFiles for: hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t1/b982e41c556da4de25932dd66dee6ea7/f1 2015-10-16 10:59:15,046 DEBUG [RS_OPEN_REGION-c2021:16020-2] regionserver.HRegion: Found 0 recovered edits file(s) under hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t1/b982e41c556da4de25932dd66dee6ea7 2015-10-16 10:59:15,050 DEBUG [RS_OPEN_REGION-c2021:16020-2] regionserver.FlushLargeStoresPolicy: hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-10-16 10:59:15,059 DEBUG [RS_OPEN_REGION-c2021:16020-1] wal.WALSplitter: Wrote region seqId=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t1/38cc05fd57524229b6809366776d6542/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-10-16 10:59:15,059 INFO [RS_OPEN_REGION-c2021:16020-1] regionserver.HRegion: Onlined 38cc05fd57524229b6809366776d6542; next sequenceid=2 2015-10-16 10:59:15,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629666_1099518554296] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 9 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 764572, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 10:59:15,063 INFO [PostOpenDeployTasks:38cc05fd57524229b6809366776d6542] regionserver.HRegionServer: Post open deploy tasks for t1,,1445018354106.38cc05fd57524229b6809366776d6542. 2015-10-16 10:59:15,071 DEBUG [PostOpenDeployTasks:38cc05fd57524229b6809366776d6542] regionserver.HRegionServer: Finished post open deploy task for t1,,1445018354106.38cc05fd57524229b6809366776d6542. 2015-10-16 10:59:15,072 DEBUG [RS_OPEN_REGION-c2021:16020-1] handler.OpenRegionHandler: Opened t1,,1445018354106.38cc05fd57524229b6809366776d6542. on c2021.halxg.cloudera.com,16020,1445018072095 2015-10-16 10:59:15,075 DEBUG [RS_OPEN_REGION-c2021:16020-2] wal.WALSplitter: Wrote region seqId=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t1/b982e41c556da4de25932dd66dee6ea7/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-10-16 10:59:15,076 INFO [RS_OPEN_REGION-c2021:16020-2] regionserver.HRegion: Onlined b982e41c556da4de25932dd66dee6ea7; next sequenceid=2 2015-10-16 10:59:15,079 INFO [PostOpenDeployTasks:b982e41c556da4de25932dd66dee6ea7] regionserver.HRegionServer: Post open deploy tasks for t1,40,1445018354106.b982e41c556da4de25932dd66dee6ea7. 2015-10-16 10:59:15,084 DEBUG [PostOpenDeployTasks:b982e41c556da4de25932dd66dee6ea7] regionserver.HRegionServer: Finished post open deploy task for t1,40,1445018354106.b982e41c556da4de25932dd66dee6ea7. 2015-10-16 10:59:15,085 DEBUG [RS_OPEN_REGION-c2021:16020-2] handler.OpenRegionHandler: Opened t1,40,1445018354106.b982e41c556da4de25932dd66dee6ea7. on c2021.halxg.cloudera.com,16020,1445018072095 2015-10-16 10:59:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0 2015-10-16 10:59:37,352 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 10:59:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:00:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:01:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:02:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:03:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:04:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0 2015-10-16 11:04:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 11:04:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:05:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:06:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:07:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:08:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:09:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=89, evicted=0, evictedPerRun=0.0 2015-10-16 11:09:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 11:09:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:10:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:11:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:12:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:13:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:14:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0 2015-10-16 11:14:37,352 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 11:14:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:15:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:16:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:17:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:18:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:19:12,983 DEBUG [RpcServer.reader=2,bindAddress=c2021.halxg.cloudera.com,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 10.20.84.26:41989 because read count=-1. Number of active connections: 2 2015-10-16 11:19:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=149, evicted=0, evictedPerRun=0.0 2015-10-16 11:19:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 11:19:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:20:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:21:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:22:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:23:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:24:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=179, evicted=0, evictedPerRun=0.0 2015-10-16 11:24:37,352 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 11:24:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:25:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:26:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:27:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:28:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:29:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=209, evicted=0, evictedPerRun=0.0 2015-10-16 11:29:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 11:29:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:30:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:31:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:32:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:33:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:34:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=239, evicted=0, evictedPerRun=0.0 2015-10-16 11:34:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 11:34:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:35:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:36:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:37:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:38:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:39:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=269, evicted=0, evictedPerRun=0.0 2015-10-16 11:39:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 11:39:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:40:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:41:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:42:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:43:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:44:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=299, evicted=0, evictedPerRun=0.0 2015-10-16 11:44:37,352 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 11:44:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:45:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:46:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:47:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:48:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:49:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=329, evicted=0, evictedPerRun=0.0 2015-10-16 11:49:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 11:49:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:50:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:51:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:52:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:53:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:54:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=359, evicted=0, evictedPerRun=0.0 2015-10-16 11:54:37,352 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 11:54:37,357 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2015-10-16 11:54:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:54:39,674 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: Wal roll period 3600000ms elapsed 2015-10-16 11:54:39,727 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629666_1099518554296] hdfs.DFSClient: Slow ReadProcessor read fields took 24608ms (threshold=20ms); ack: seqno: 12 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905996, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 11:54:39,738 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445018082849 with entries=10, filesize=1.86 KB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445021679674 2015-10-16 11:55:19,625 INFO [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] regionserver.HRegionServer: c2021.halxg.cloudera.com,16020,1445018072095-MemstoreFlusherChore requesting flush of t,,1445018090873.07b76eb51213346d939cd30591ffe6d2. because t has an old edit so flush to free WALs after random delay 9726ms 2015-10-16 11:55:29,356 DEBUG [MemStoreFlusher.0] regionserver.HRegion: Flush column family: t of 07b76eb51213346d939cd30591ffe6d2 because time of oldest edit=1445018112136 is > 3600000 from now =1445021729356 2015-10-16 11:55:29,357 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=632 B 2015-10-16 11:55:29,376 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 19660ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924344, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 11:55:29,490 WARN [MemStoreFlusher.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 11:55:29,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=9, memsize=632, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t/07b76eb51213346d939cd30591ffe6d2/.tmp/c8e86dffe53a40eeaf29a7813da5f1fd 2015-10-16 11:55:29,572 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t/07b76eb51213346d939cd30591ffe6d2/.tmp/c8e86dffe53a40eeaf29a7813da5f1fd as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t/07b76eb51213346d939cd30591ffe6d2/t/c8e86dffe53a40eeaf29a7813da5f1fd 2015-10-16 11:55:29,594 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/t/07b76eb51213346d939cd30591ffe6d2/t/c8e86dffe53a40eeaf29a7813da5f1fd, entries=3, sequenceid=9, filesize=4.8 K 2015-10-16 11:55:29,598 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 222ms (threshold=20ms); ack: seqno: 2 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835028, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 11:55:29,599 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~632 B/632, currentsize=0 B/0 for region t,,1445018090873.07b76eb51213346d939cd30591ffe6d2. in 242ms, sequenceid=9, compaction requested=false 2015-10-16 11:55:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:56:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:57:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:58:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 11:59:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=389, evicted=0, evictedPerRun=0.0 2015-10-16 11:59:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 11:59:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:00:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:01:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:02:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:03:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:04:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=419, evicted=0, evictedPerRun=0.0 2015-10-16 12:04:37,352 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 12:04:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:05:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:06:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:07:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:08:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:09:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=449, evicted=0, evictedPerRun=0.0 2015-10-16 12:09:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 12:09:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:10:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:11:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:12:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:13:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:14:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=479, evicted=0, evictedPerRun=0.0 2015-10-16 12:14:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 12:14:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:15:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:16:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:17:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:18:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:19:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=509, evicted=0, evictedPerRun=0.0 2015-10-16 12:19:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 12:19:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:20:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:21:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:22:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:23:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:24:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=539, evicted=0, evictedPerRun=0.0 2015-10-16 12:24:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 12:24:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:25:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:26:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:27:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:28:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:29:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=569, evicted=0, evictedPerRun=0.0 2015-10-16 12:29:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 12:29:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:30:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:31:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:32:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:33:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:34:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=599, evicted=0, evictedPerRun=0.0 2015-10-16 12:34:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 12:34:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:35:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:36:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:37:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:38:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:39:32,127 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=629, evicted=0, evictedPerRun=0.0 2015-10-16 12:39:37,352 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=0, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 12:39:38,103 INFO [PriorityRpcServer.handler=19,queue=1,port=16020] regionserver.RSRpcServices: Open ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:39:38,119 DEBUG [RS_OPEN_REGION-c2021:16020-0] regionserver.HRegion: Opening region: {ENCODED => 127e05e2659db66008aa963b54da70f6, NAME => 'ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.', STARTKEY => '', ENDKEY => ''} 2015-10-16 12:39:38,120 DEBUG [RS_OPEN_REGION-c2021:16020-0] regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl for table ycsb 127e05e2659db66008aa963b54da70f6 2015-10-16 12:39:38,120 DEBUG [RS_OPEN_REGION-c2021:16020-0] regionserver.HRegion: Instantiated ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:39:38,129 INFO [StoreOpener-127e05e2659db66008aa963b54da70f6-1] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:39:38,129 INFO [StoreOpener-127e05e2659db66008aa963b54da70f6-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-10-16 12:39:38,131 DEBUG [StoreOpener-127e05e2659db66008aa963b54da70f6-1] regionserver.HRegionFileSystem: No StoreFiles for: hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family 2015-10-16 12:39:38,133 DEBUG [RS_OPEN_REGION-c2021:16020-0] regionserver.HRegion: Found 0 recovered edits file(s) under hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6 2015-10-16 12:39:38,136 DEBUG [RS_OPEN_REGION-c2021:16020-0] regionserver.FlushLargeStoresPolicy: hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-10-16 12:39:38,154 DEBUG [RS_OPEN_REGION-c2021:16020-0] wal.WALSplitter: Wrote region seqId=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-10-16 12:39:38,154 INFO [RS_OPEN_REGION-c2021:16020-0] regionserver.HRegion: Onlined 127e05e2659db66008aa963b54da70f6; next sequenceid=2 2015-10-16 12:39:38,157 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 8530ms (threshold=20ms); ack: seqno: 3 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838443, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:39:38,158 INFO [PostOpenDeployTasks:127e05e2659db66008aa963b54da70f6] regionserver.HRegionServer: Post open deploy tasks for ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:39:38,163 DEBUG [PostOpenDeployTasks:127e05e2659db66008aa963b54da70f6] regionserver.HRegionServer: Finished post open deploy task for ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:39:38,164 DEBUG [RS_OPEN_REGION-c2021:16020-0] handler.OpenRegionHandler: Opened ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. on c2021.halxg.cloudera.com,16020,1445018072095 2015-10-16 12:39:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2015-10-16 12:39:56,626 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43220; # active connections: 2 2015-10-16 12:39:56,628 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43221; # active connections: 3 2015-10-16 12:39:56,628 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43223; # active connections: 4 2015-10-16 12:39:56,629 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43222; # active connections: 5 2015-10-16 12:39:56,632 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43224; # active connections: 6 2015-10-16 12:39:56,632 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43225; # active connections: 7 2015-10-16 12:39:56,633 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43226; # active connections: 8 2015-10-16 12:39:56,633 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43227; # active connections: 9 2015-10-16 12:39:56,635 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43228; # active connections: 10 2015-10-16 12:39:56,635 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43229; # active connections: 11 2015-10-16 12:39:56,638 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43230; # active connections: 12 2015-10-16 12:39:56,639 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43231; # active connections: 13 2015-10-16 12:39:56,639 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43232; # active connections: 14 2015-10-16 12:39:56,640 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43233; # active connections: 15 2015-10-16 12:39:56,644 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43234; # active connections: 16 2015-10-16 12:39:56,645 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43235; # active connections: 17 2015-10-16 12:39:56,648 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43236; # active connections: 18 2015-10-16 12:39:56,650 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43237; # active connections: 19 2015-10-16 12:39:56,651 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43238; # active connections: 20 2015-10-16 12:39:56,651 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43239; # active connections: 21 2015-10-16 12:39:56,651 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43240; # active connections: 22 2015-10-16 12:39:56,653 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43241; # active connections: 23 2015-10-16 12:39:56,654 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43242; # active connections: 24 2015-10-16 12:39:56,655 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43243; # active connections: 25 2015-10-16 12:39:56,656 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43245; # active connections: 26 2015-10-16 12:39:56,657 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43244; # active connections: 27 2015-10-16 12:39:56,658 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43246; # active connections: 28 2015-10-16 12:39:56,658 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43248; # active connections: 29 2015-10-16 12:39:56,659 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43247; # active connections: 30 2015-10-16 12:39:56,659 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.20.84.26:43249; # active connections: 31 2015-10-16 12:39:56,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 18510ms (threshold=20ms); ack: seqno: 4 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1590853, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:39:56,707 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 16 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808070, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:39:57,627 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 1773 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 607688, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:39:58,695 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 3208 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 600891, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:39:58,756 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 3229 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 574122, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:39:58,812 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 3251 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 589413, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:39:58,862 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 3261 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 661722, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:00,770 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 5000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 571313, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:00,770 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:00,960 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 5150 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 694795, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:01,862 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 193ms (threshold=20ms); ack: seqno: 5727 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 938207, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:01,862 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 188ms (threshold=20ms) 2015-10-16 12:40:01,862 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 192ms (threshold=20ms) 2015-10-16 12:40:01,865 INFO [sync.0] wal.FSHLog: Slow sync cost: 189 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:01,864 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 195ms (threshold=20ms) 2015-10-16 12:40:01,862 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 191ms (threshold=20ms) 2015-10-16 12:40:01,865 INFO [sync.4] wal.FSHLog: Slow sync cost: 193 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:01,862 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 191ms (threshold=20ms) 2015-10-16 12:40:01,865 INFO [sync.1] wal.FSHLog: Slow sync cost: 196 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:01,865 INFO [sync.2] wal.FSHLog: Slow sync cost: 194 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:01,865 INFO [sync.3] wal.FSHLog: Slow sync cost: 193 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:01,986 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 5810 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 665826, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:03,088 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 6581 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 635353, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:04,116 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 7319 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 712658, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:04,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 8504 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 705569, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:05,076 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 11142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 501955, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:05,076 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:05,076 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:05,076 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:05,076 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:05,076 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:05,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 13929 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 417341, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:06,079 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:40:06,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 16509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 490593, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:06,080 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:06,080 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:06,080 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:40:06,627 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 19518 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 464766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:06,954 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 20778 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 524412, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:07,199 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 22013 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 442898, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:07,200 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:40:07,200 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:07,201 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:40:07,201 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:40:07,201 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:08,076 DEBUG [PriorityRpcServer.handler=7,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:40:08,077 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134290896 > lower bound=16777216 2015-10-16 12:40:08,077 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.04 MB 2015-10-16 12:40:08,106 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 27049 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 472579, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:08,239 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 27390 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 527913, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:08,240 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:40:08,240 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:08,240 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:40:08,240 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:40:08,241 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:40:08,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629685_1099518554315] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 18 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2191360, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:40:08,978 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629685_1099518554315] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 940 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 772565, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:40:08,978 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 30883 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 566677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:08,979 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:08,979 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:08,979 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:08,979 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:08,979 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:09,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=46649, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a63fe748f96349b8a615686b1735fbdb 2015-10-16 12:40:09,124 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a63fe748f96349b8a615686b1735fbdb as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a63fe748f96349b8a615686b1735fbdb 2015-10-16 12:40:09,134 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a63fe748f96349b8a615686b1735fbdb, entries=466440, sequenceid=46649, filesize=70.4 M 2015-10-16 12:40:09,135 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.11 MB/134330800, currentsize=26.86 MB/28168880 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1058ms, sequenceid=46649, compaction requested=false 2015-10-16 12:40:09,554 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 33660 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 70023400, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:09,555 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 12:40:09,555 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:40:09,555 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:40:09,555 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 12:40:09,555 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:40:10,378 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 641ms (threshold=20ms); ack: seqno: 34630 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 554717868, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:10,378 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 556ms (threshold=20ms) 2015-10-16 12:40:10,378 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 555ms (threshold=20ms) 2015-10-16 12:40:10,379 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 556ms (threshold=20ms) 2015-10-16 12:40:10,379 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 556ms (threshold=20ms) 2015-10-16 12:40:10,379 INFO [sync.2] wal.FSHLog: Slow sync cost: 556 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:10,379 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 556ms (threshold=20ms) 2015-10-16 12:40:10,379 INFO [sync.0] wal.FSHLog: Slow sync cost: 555 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:10,379 INFO [sync.1] wal.FSHLog: Slow sync cost: 556 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:10,380 INFO [sync.3] wal.FSHLog: Slow sync cost: 556 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:10,379 INFO [sync.4] wal.FSHLog: Slow sync cost: 556 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:10,646 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 35957 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 622194, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:10,647 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:40:10,647 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:40:10,647 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:40:10,647 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:40:10,647 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:40:11,637 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629678_1099518554308] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 41524 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489361, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:11,638 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:40:11,638 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:40:11,638 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:40:11,638 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:40:11,638 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:40:11,790 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:40:11,827 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:11,844 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445021679674 with entries=78130, filesize=122.38 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024411790 2015-10-16 12:40:11,846 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445018082849 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445018082849 2015-10-16 12:40:12,621 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629686_1099518554316] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 4303 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 461854, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:40:12,621 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:12,621 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:12,621 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:12,621 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:12,621 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:13,203 DEBUG [PriorityRpcServer.handler=11,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:40:13,204 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134274096 > lower bound=16777216 2015-10-16 12:40:13,204 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:40:13,229 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629686_1099518554316] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 7590 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 668797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:40:13,568 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629686_1099518554316] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 8994 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 633752, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:40:13,568 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:40:13,568 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:40:13,569 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:40:13,569 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:40:13,569 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:40:13,569 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629687_1099518554317] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 403 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 689214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:40:14,093 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=93284, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/adb5806ad16b496095825f5575137184 2015-10-16 12:40:14,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/adb5806ad16b496095825f5575137184 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/adb5806ad16b496095825f5575137184 2015-10-16 12:40:14,115 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/adb5806ad16b496095825f5575137184, entries=466310, sequenceid=93284, filesize=70.7 M 2015-10-16 12:40:14,117 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134293840, currentsize=23.63 MB/24781520 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 913ms, sequenceid=93284, compaction requested=false 2015-10-16 12:40:14,339 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629686_1099518554316] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 13035 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 485920, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:40:14,340 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:14,340 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:14,340 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:14,340 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:14,340 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:15,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629686_1099518554316] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 18608 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 427385, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:40:15,280 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:15,280 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:15,280 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:15,280 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:15,280 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:15,854 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629686_1099518554316] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 21828 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 476508, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:40:16,232 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629686_1099518554316] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 23515 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 532226, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:40:16,375 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629686_1099518554316] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 24154 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 436588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:40:16,375 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:16,375 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:40:17,379 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629686_1099518554316] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 29717 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 611051, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:40:17,379 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:17,379 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:17,379 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:17,379 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:17,379 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:17,617 DEBUG [PriorityRpcServer.handler=5,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:40:17,617 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134270736 > lower bound=16777216 2015-10-16 12:40:17,634 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.60 MB 2015-10-16 12:40:17,656 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629686_1099518554316] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 31208 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 587707, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:40:18,147 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629688_1099518554318] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 811 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 742575, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:18,147 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629686_1099518554316] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 33640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 522946, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:40:18,149 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:18,149 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:18,149 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:18,149 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:18,149 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:18,368 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=140130, memsize=128.7 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/30af564eb9ff46809b6f99b391b60666 2015-10-16 12:40:18,379 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/30af564eb9ff46809b6f99b391b60666 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/30af564eb9ff46809b6f99b391b60666 2015-10-16 12:40:18,399 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/30af564eb9ff46809b6f99b391b60666, entries=468420, sequenceid=140130, filesize=71.2 M 2015-10-16 12:40:18,401 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.65 MB/134901040, currentsize=20.88 MB/21890560 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 767ms, sequenceid=140130, compaction requested=true 2015-10-16 12:40:18,403 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:40:18,404 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 3 store files, 0 compacting, 3 eligible, 10 blocking 2015-10-16 12:40:18,407 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222543614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 12:40:18,409 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction (all files) 2015-10-16 12:40:18,410 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:40:18,410 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.2 M 2015-10-16 12:40:18,411 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a63fe748f96349b8a615686b1735fbdb, keycount=466440, bloomtype=ROW, size=70.4 M, encoding=NONE, seqNum=46649, earliestPutTs=1445024396631 2015-10-16 12:40:18,412 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/adb5806ad16b496095825f5575137184, keycount=466310, bloomtype=ROW, size=70.7 M, encoding=NONE, seqNum=93284, earliestPutTs=1445024408078 2015-10-16 12:40:18,413 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/30af564eb9ff46809b6f99b391b60666, keycount=468420, bloomtype=ROW, size=71.2 M, encoding=NONE, seqNum=140130, earliestPutTs=1445024413204 2015-10-16 12:40:18,442 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:40:18,998 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:40:18,998 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:40:18,998 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:40:18,998 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:40:18,998 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:40:19,207 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:40:19,260 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024411790 with entries=78023, filesize=122.23 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024419207 2015-10-16 12:40:19,262 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445021679674 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445021679674 2015-10-16 12:40:19,263 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629690_1099518554320] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 970714, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:19,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 575ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1790632, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:19,854 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629690_1099518554320] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 3391 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 438697, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:19,854 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:40:19,854 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:40:19,854 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:40:19,854 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:40:20,453 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 831ms (threshold=20ms); ack: seqno: 331 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1197174, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:20,669 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 483 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794799, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:20,669 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629690_1099518554320] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 8290 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 567882, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:20,670 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:40:20,670 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:40:20,670 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:40:20,670 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:40:20,670 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:40:21,329 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 81ms (threshold=20ms) 2015-10-16 12:40:21,329 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629690_1099518554320] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 11951 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 448561, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:21,329 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 82ms (threshold=20ms) 2015-10-16 12:40:21,329 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 82ms (threshold=20ms) 2015-10-16 12:40:21,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 763ms (threshold=20ms); ack: seqno: 498 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1402345, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:21,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 592 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782666, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:21,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629690_1099518554320] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 13356 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 410196, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:21,577 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:21,577 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:21,577 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:21,578 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:22,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629690_1099518554320] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 15416 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 582939, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:22,009 DEBUG [PriorityRpcServer.handler=17,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:40:22,009 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134273856 > lower bound=16777216 2015-10-16 12:40:22,009 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:40:22,032 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629690_1099518554320] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 15465 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 514196, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:22,428 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:22,428 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629690_1099518554320] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 17548 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 485203, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:22,428 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629691_1099518554321] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 520 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834821, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:22,428 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:22,428 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:22,428 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:22,428 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:22,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 809ms (threshold=20ms); ack: seqno: 664 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1380177, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:22,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=186763, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/16c7103ecf9849228b4a4ee8e4f36e3b 2015-10-16 12:40:22,876 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/16c7103ecf9849228b4a4ee8e4f36e3b as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/16c7103ecf9849228b4a4ee8e4f36e3b 2015-10-16 12:40:22,889 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/16c7103ecf9849228b4a4ee8e4f36e3b, entries=466290, sequenceid=186763, filesize=70.8 M 2015-10-16 12:40:22,890 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134287840, currentsize=25.16 MB/26382800 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 881ms, sequenceid=186763, compaction requested=false 2015-10-16 12:40:23,084 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629690_1099518554320] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 21209 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 421796, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:23,084 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:23,084 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:23,084 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:23,084 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:23,084 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:23,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 842ms (threshold=20ms); ack: seqno: 831 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1426530, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:23,901 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629690_1099518554320] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 26031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 410691, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:23,902 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:23,902 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:23,902 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:23,902 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:23,902 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:24,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 850ms (threshold=20ms); ack: seqno: 997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1426974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:24,726 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629690_1099518554320] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 30919 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 456000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:24,726 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:24,726 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:24,726 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:24,726 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:24,726 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:25,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 853ms (threshold=20ms); ack: seqno: 1164 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1440432, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:25,572 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629690_1099518554320] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 36046 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 574698, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:25,572 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1260 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 876937, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:25,572 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:40:25,572 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:40:25,572 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:40:25,572 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:40:25,572 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:40:26,043 DEBUG [PriorityRpcServer.handler=16,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:40:26,043 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134284736 > lower bound=16777216 2015-10-16 12:40:26,044 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:40:26,070 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629690_1099518554320] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 39072 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 479813, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:26,103 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:40:26,134 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024419207 with entries=77844, filesize=121.95 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024426103 2015-10-16 12:40:26,136 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024411790 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024411790 2015-10-16 12:40:26,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629693_1099518554323] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 1126 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 458122, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:26,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629692_1099518554322] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 377 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 753126, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:26,372 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:26,372 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:26,372 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:26,372 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:26,372 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:26,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 818ms (threshold=20ms); ack: seqno: 1330 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1179502, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:26,942 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=233395, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/9fd933f5ae4f46ddbaa0016a0c5d73cc 2015-10-16 12:40:26,950 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/9fd933f5ae4f46ddbaa0016a0c5d73cc as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9fd933f5ae4f46ddbaa0016a0c5d73cc 2015-10-16 12:40:26,964 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9fd933f5ae4f46ddbaa0016a0c5d73cc, entries=466280, sequenceid=233395, filesize=70.8 M 2015-10-16 12:40:26,966 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134284320, currentsize=25.61 MB/26849840 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 923ms, sequenceid=233395, compaction requested=false 2015-10-16 12:40:27,006 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629693_1099518554323] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 4464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 431807, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:27,006 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:27,006 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:27,006 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:27,006 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:27,006 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:27,032 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629693_1099518554323] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 4481 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 529309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:27,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 852ms (threshold=20ms); ack: seqno: 1496 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:27,839 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629693_1099518554323] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 9356 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 483435, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:27,839 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:27,839 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:27,839 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:27,839 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:27,839 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:28,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 852ms (threshold=20ms); ack: seqno: 1662 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1248830, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:28,704 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629693_1099518554323] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 14320 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 521292, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:28,704 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:28,704 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:28,704 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:28,704 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:28,704 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:29,456 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 853ms (threshold=20ms); ack: seqno: 1829 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2847787, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:29,557 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629693_1099518554323] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 19536 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 464037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:29,557 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 1924 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4913008, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:29,558 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:40:29,557 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:40:29,557 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:40:29,558 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:30,170 DEBUG [PriorityRpcServer.handler=14,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:40:30,170 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134276528 > lower bound=16777216 2015-10-16 12:40:30,170 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:40:30,196 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629693_1099518554323] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 23416 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 472119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:30,404 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629693_1099518554323] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 24371 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 506787, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:30,404 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629694_1099518554324] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964577, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:30,404 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:30,404 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:40:30,404 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:40:30,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629689_1099518554319] hdfs.DFSClient: Slow ReadProcessor read fields took 835ms (threshold=20ms); ack: seqno: 1995 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207480, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:30,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629694_1099518554324] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 667 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:30,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629693_1099518554323] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 26554 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 505713, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:30,873 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 67ms (threshold=20ms) 2015-10-16 12:40:30,873 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 68ms (threshold=20ms) 2015-10-16 12:40:30,873 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 68ms (threshold=20ms) 2015-10-16 12:40:30,873 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 68ms (threshold=20ms) 2015-10-16 12:40:30,873 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 68ms (threshold=20ms) 2015-10-16 12:40:31,106 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629693_1099518554323] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 27736 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 398720, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:31,106 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629694_1099518554324] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 966 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 780431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:31,106 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:31,106 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:31,106 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:31,251 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=280033, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/cdd51822d73740e9b9ddbe442388e907 2015-10-16 12:40:31,259 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/cdd51822d73740e9b9ddbe442388e907 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cdd51822d73740e9b9ddbe442388e907 2015-10-16 12:40:31,280 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cdd51822d73740e9b9ddbe442388e907, entries=466340, sequenceid=280033, filesize=70.8 M 2015-10-16 12:40:31,282 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134302320, currentsize=29.21 MB/30633520 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1112ms, sequenceid=280033, compaction requested=true 2015-10-16 12:40:31,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:40:31,284 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 6 store files, 3 compacting, 3 eligible, 10 blocking 2015-10-16 12:40:31,286 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222842665 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 12:40:31,287 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:40:31,287 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:40:31,288 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 12:40:31,288 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/16c7103ecf9849228b4a4ee8e4f36e3b, keycount=466290, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=186763 2015-10-16 12:40:31,289 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9fd933f5ae4f46ddbaa0016a0c5d73cc, keycount=466280, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=233395 2015-10-16 12:40:31,289 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cdd51822d73740e9b9ddbe442388e907, keycount=466340, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=280033 2015-10-16 12:40:31,313 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:40:31,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629695_1099518554325] hdfs.DFSClient: Slow ReadProcessor read fields took 802ms (threshold=20ms); ack: seqno: 2165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1246010, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:40:31,788 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629693_1099518554323] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 31520 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 481671, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:31,788 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:31,788 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:31,789 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:31,789 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:31,789 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:31,883 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629693_1099518554323] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 31632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 550168, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:32,741 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629693_1099518554323] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 37063 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911533, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:32,741 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:32,741 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:32,742 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:32,742 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:32,742 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:32,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629693_1099518554323] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 38437 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:33,135 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:40:33,184 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024426103 with entries=78124, filesize=122.39 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024433135 2015-10-16 12:40:33,185 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024419207 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024419207 2015-10-16 12:40:33,320 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629696_1099518554326] hdfs.DFSClient: Slow ReadProcessor read fields took 1859ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1379729, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:33,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629695_1099518554325] hdfs.DFSClient: Slow ReadProcessor read fields took 1868ms (threshold=20ms); ack: seqno: 2333 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1091720, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:40:33,562 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629697_1099518554327] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 2006 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 538227, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:33,562 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629695_1099518554325] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 2411 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751712, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:40:33,562 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:33,562 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:33,562 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:33,563 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:33,562 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:34,445 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629697_1099518554327] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 7172 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 418789, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:34,445 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:40:34,445 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:40:34,445 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:40:34,445 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:40:34,445 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:40:34,475 DEBUG [PriorityRpcServer.handler=4,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:40:34,475 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134281856 > lower bound=16777216 2015-10-16 12:40:34,475 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:40:34,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629698_1099518554328] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 490 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 778971, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:34,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629697_1099518554327] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 9354 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 407808, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:34,913 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:40:34,913 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 12:40:34,913 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:40:34,913 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 12:40:35,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629697_1099518554327] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 10971 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 457642, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:35,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629698_1099518554328] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 892 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 745406, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:35,215 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:35,215 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:35,215 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:35,215 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:35,215 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:35,321 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629696_1099518554326] hdfs.DFSClient: Slow ReadProcessor read fields took 1866ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1385813, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:35,400 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=326665, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c2e11276b1544ef580a311f1987f4d3b 2015-10-16 12:40:35,408 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c2e11276b1544ef580a311f1987f4d3b as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c2e11276b1544ef580a311f1987f4d3b 2015-10-16 12:40:35,421 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c2e11276b1544ef580a311f1987f4d3b, entries=466280, sequenceid=326665, filesize=70.8 M 2015-10-16 12:40:35,422 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134284320, currentsize=26.62 MB/27911600 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 947ms, sequenceid=326665, compaction requested=false 2015-10-16 12:40:35,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629695_1099518554325] hdfs.DFSClient: Slow ReadProcessor read fields took 1817ms (threshold=20ms); ack: seqno: 2500 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037194, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:40:35,582 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629695_1099518554325] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 2626 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 20226145, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:40:35,857 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629697_1099518554327] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 14651 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 425592, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:35,857 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:35,857 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:35,857 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:35,857 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:35,857 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:35,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629697_1099518554327] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 14805 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 551261, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:36,788 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:36,788 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:36,788 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:36,788 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:36,788 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629697_1099518554327] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 20206 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 463630, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:36,788 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:37,306 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629697_1099518554327] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 23229 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 556513, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:37,335 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629696_1099518554326] hdfs.DFSClient: Slow ReadProcessor read fields took 1881ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 15891018, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:37,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629695_1099518554325] hdfs.DFSClient: Slow ReadProcessor read fields took 1859ms (threshold=20ms); ack: seqno: 2666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111523, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:40:37,588 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629697_1099518554327] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 24545 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 510017, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:37,588 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629695_1099518554325] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 2771 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 770312, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:40:37,588 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:40:37,589 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:40:37,588 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:40:37,589 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:40:37,589 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:40:38,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629697_1099518554327] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 27908 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 403256, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:38,212 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 79ms (threshold=20ms) 2015-10-16 12:40:38,212 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 79ms (threshold=20ms) 2015-10-16 12:40:38,212 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 79ms (threshold=20ms) 2015-10-16 12:40:38,212 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 79ms (threshold=20ms) 2015-10-16 12:40:38,212 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 78ms (threshold=20ms) 2015-10-16 12:40:38,552 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629697_1099518554327] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 29849 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 392795, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:38,552 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:38,552 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:38,552 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:38,552 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:38,553 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:38,712 DEBUG [PriorityRpcServer.handler=14,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:40:38,712 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134282496 > lower bound=16777216 2015-10-16 12:40:38,743 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=129.05 MB 2015-10-16 12:40:38,769 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629697_1099518554327] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 31047 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 484126, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.5714285714285714, tune compaction throughput to 15.71 MB/sec 2015-10-16 12:40:39,315 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:39,316 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629697_1099518554327] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 33885 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 419780, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:39,316 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629699_1099518554329] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 749 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 796935, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:39,316 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:39,316 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:39,316 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:39,316 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:39,321 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629696_1099518554326] hdfs.DFSClient: Slow ReadProcessor read fields took 1857ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1136452, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:39,456 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629695_1099518554325] hdfs.DFSClient: Slow ReadProcessor read fields took 1803ms (threshold=20ms); ack: seqno: 2835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1288592, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:40:39,666 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=373676, memsize=129.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b1993562eff143109b9a4148f892a2ba 2015-10-16 12:40:39,673 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b1993562eff143109b9a4148f892a2ba as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b1993562eff143109b9a4148f892a2ba 2015-10-16 12:40:39,688 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b1993562eff143109b9a4148f892a2ba, entries=470070, sequenceid=373676, filesize=71.4 M 2015-10-16 12:40:39,690 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~129.10 MB/135376400, currentsize=25.72 MB/26970240 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 947ms, sequenceid=373676, compaction requested=false 2015-10-16 12:40:39,977 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629697_1099518554327] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 37188 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844090, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:39,977 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:39,977 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:39,977 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:39,977 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:39,977 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:40,288 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:40:40,333 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024433135 with entries=78073, filesize=122.31 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024440289 2015-10-16 12:40:40,335 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024426103 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024426103 2015-10-16 12:40:40,336 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629700_1099518554330] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 935608, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:40,593 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629696_1099518554326] hdfs.DFSClient: Slow ReadProcessor read fields took 1126ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1106182, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:40,727 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629695_1099518554325] hdfs.DFSClient: Slow ReadProcessor read fields took 1091ms (threshold=20ms); ack: seqno: 3001 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1224488, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:40:40,798 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629700_1099518554330] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 2557 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 395031, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:40,798 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629695_1099518554325] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 3059 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 708800, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:40:40,798 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:40,798 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:40,798 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:41,276 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 78ms (threshold=20ms) 2015-10-16 12:40:41,276 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629700_1099518554330] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 4986 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 521795, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:41,276 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 78ms (threshold=20ms) 2015-10-16 12:40:41,276 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 78ms (threshold=20ms) 2015-10-16 12:40:41,276 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 78ms (threshold=20ms) 2015-10-16 12:40:41,276 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 78ms (threshold=20ms) 2015-10-16 12:40:41,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629700_1099518554330] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 7651 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 355446, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:41,735 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:41,865 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629696_1099518554326] hdfs.DFSClient: Slow ReadProcessor read fields took 1136ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1034916, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:42,000 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629695_1099518554325] hdfs.DFSClient: Slow ReadProcessor read fields took 1114ms (threshold=20ms); ack: seqno: 3168 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1244555, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:40:42,183 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629700_1099518554330] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 9812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 619283, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:42,554 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629700_1099518554330] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 11845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 391033, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:42,554 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:42,554 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:42,555 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:43,100 DEBUG [PriorityRpcServer.handler=19,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:40:43,101 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134284352 > lower bound=16777216 2015-10-16 12:40:43,101 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:40:43,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629700_1099518554330] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 15202 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 589311, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:43,138 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629696_1099518554326] hdfs.DFSClient: Slow ReadProcessor read fields took 1132ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1270889, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:43,271 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629695_1099518554325] hdfs.DFSClient: Slow ReadProcessor read fields took 1149ms (threshold=20ms); ack: seqno: 3334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174134, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:40:43,335 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:43,335 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629700_1099518554330] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 16007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 426355, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:43,335 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629701_1099518554331] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 232 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 890904, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:43,335 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629695_1099518554325] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 3377 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813450, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:40:43,335 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:43,335 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:43,335 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:40:43,393 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#0 average throughput is 8.30 MB/sec, slept 20 time(s) and total slept time is 21201 ms. 1 active compactions remaining, total limit is 15.71 MB/sec 2015-10-16 12:40:43,416 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/4463fa30fe63407d9f7adf901c26a347 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4463fa30fe63407d9f7adf901c26a347 2015-10-16 12:40:43,441 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:40:43,458 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:40:43,466 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a63fe748f96349b8a615686b1735fbdb, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a63fe748f96349b8a615686b1735fbdb 2015-10-16 12:40:43,474 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/adb5806ad16b496095825f5575137184, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/adb5806ad16b496095825f5575137184 2015-10-16 12:40:43,483 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/30af564eb9ff46809b6f99b391b60666, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/30af564eb9ff46809b6f99b391b60666 2015-10-16 12:40:43,483 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 (all) file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 4463fa30fe63407d9f7adf901c26a347(size=212.2 M), total size for store is 567.0 M. This selection was in queue for 0sec, and took 25sec to execute. 2015-10-16 12:40:43,486 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.2 M, priority=7, time=36902327263486214; duration=25sec 2015-10-16 12:40:43,486 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:40:43,994 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:43,994 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:43,994 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629701_1099518554331] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1088 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 732441, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:40:44,041 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=420311, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/6a3b7e8d9b114a74ba2f7ce53df7b9aa 2015-10-16 12:40:44,047 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/6a3b7e8d9b114a74ba2f7ce53df7b9aa as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6a3b7e8d9b114a74ba2f7ce53df7b9aa 2015-10-16 12:40:44,071 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6a3b7e8d9b114a74ba2f7ce53df7b9aa, entries=466310, sequenceid=420311, filesize=70.8 M 2015-10-16 12:40:44,073 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134293440, currentsize=26.84 MB/28148640 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 972ms, sequenceid=420311, compaction requested=true 2015-10-16 12:40:44,073 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 3 compacting, 3 eligible, 10 blocking 2015-10-16 12:40:44,073 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:40:44,075 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 223440101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 12:40:44,075 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:40:44,075 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:40:44,075 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=213.1 M 2015-10-16 12:40:44,075 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c2e11276b1544ef580a311f1987f4d3b, keycount=466280, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=326665 2015-10-16 12:40:44,076 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b1993562eff143109b9a4148f892a2ba, keycount=470070, bloomtype=ROW, size=71.4 M, encoding=NONE, seqNum=373676 2015-10-16 12:40:44,076 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6a3b7e8d9b114a74ba2f7ce53df7b9aa, keycount=466310, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=420311 2015-10-16 12:40:44,095 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:40:44,430 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 68ms (threshold=20ms) 2015-10-16 12:40:44,430 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 68ms (threshold=20ms) 2015-10-16 12:40:44,430 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 68ms (threshold=20ms) 2015-10-16 12:40:44,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629700_1099518554330] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 21807 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 494899, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:44,431 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 68ms (threshold=20ms) 2015-10-16 12:40:44,431 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 68ms (threshold=20ms) 2015-10-16 12:40:44,432 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629696_1099518554326] hdfs.DFSClient: Slow ReadProcessor read fields took 1136ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1132440, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:44,755 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629700_1099518554330] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 23552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 412875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:44,755 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:40:44,755 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:40:44,755 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:40:45,375 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629702_1099518554332] hdfs.DFSClient: Slow ReadProcessor read fields took 1101ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1643937, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:40:45,566 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629700_1099518554330] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 28420 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 464469, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:45,566 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:45,566 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:45,566 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:45,566 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:45,566 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:45,705 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629696_1099518554326] hdfs.DFSClient: Slow ReadProcessor read fields took 1133ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020069, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:46,409 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629700_1099518554330] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 33305 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 439308, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:46,409 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:46,409 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:46,409 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:46,409 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:46,409 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:40:46,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629702_1099518554332] hdfs.DFSClient: Slow ReadProcessor read fields took 1130ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1681836, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:40:46,977 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629696_1099518554326] hdfs.DFSClient: Slow ReadProcessor read fields took 1134ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1056663, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:47,145 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629700_1099518554330] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 37496 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 379415, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:47,146 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:47,146 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:47,146 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:47,146 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:47,146 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:47,321 DEBUG [PriorityRpcServer.handler=7,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:40:47,322 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134279056 > lower bound=16777216 2015-10-16 12:40:47,322 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:40:47,351 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629700_1099518554330] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 38606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 435945, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:40:47,426 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:40:47,467 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024440289 with entries=77967, filesize=122.14 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024447427 2015-10-16 12:40:47,468 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024433135 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024433135 2015-10-16 12:40:47,469 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629704_1099518554334] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1018666, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:47,898 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629703_1099518554333] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 725 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830589, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:40:47,898 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629704_1099518554334] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 2347 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 570066, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:47,899 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:47,899 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:47,899 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:40:47,899 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:47,919 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629702_1099518554332] hdfs.DFSClient: Slow ReadProcessor read fields took 1129ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1416957, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:40:48,208 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=466949, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/dbe4b98577ce49f4853db95776593add 2015-10-16 12:40:48,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/dbe4b98577ce49f4853db95776593add as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/dbe4b98577ce49f4853db95776593add 2015-10-16 12:40:48,228 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/dbe4b98577ce49f4853db95776593add, entries=466330, sequenceid=466949, filesize=70.8 M 2015-10-16 12:40:48,230 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134298800, currentsize=26.63 MB/27923600 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 908ms, sequenceid=466949, compaction requested=false 2015-10-16 12:40:48,252 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629696_1099518554326] hdfs.DFSClient: Slow ReadProcessor read fields took 1137ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2860150, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:48,506 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:48,506 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629704_1099518554334] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 5712 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 566025, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:48,506 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:48,506 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:40:48,506 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:40:48,506 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:40:49,061 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:40:49,061 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629704_1099518554334] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 8682 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 422062, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:49,061 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:40:49,062 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:40:49,191 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629702_1099518554332] hdfs.DFSClient: Slow ReadProcessor read fields took 1144ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1297730, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:40:49,416 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629704_1099518554334] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 10574 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 585382, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:49,416 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:49,416 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:49,416 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:49,416 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:49,416 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:49,522 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629696_1099518554326] hdfs.DFSClient: Slow ReadProcessor read fields took 1148ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1239507, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:50,243 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629704_1099518554334] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 15420 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 494438, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:50,243 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:50,243 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:50,243 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:50,243 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:50,464 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629702_1099518554332] hdfs.DFSClient: Slow ReadProcessor read fields took 1149ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1352233, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:40:50,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629705_1099518554335] hdfs.DFSClient: Slow ReadProcessor read fields took 1147ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1537575, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:50,974 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629704_1099518554334] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 19606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 492112, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:50,974 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:50,974 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:50,974 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:50,974 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:50,974 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:51,496 DEBUG [PriorityRpcServer.handler=9,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:40:51,496 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134281808 > lower bound=16777216 2015-10-16 12:40:51,497 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:40:51,525 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629704_1099518554334] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 22977 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 458436, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:51,741 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629702_1099518554332] hdfs.DFSClient: Slow ReadProcessor read fields took 1158ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7035607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:40:51,809 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629704_1099518554334] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 24316 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 442717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:51,809 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:40:51,809 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:40:51,810 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629706_1099518554336] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 333 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101295, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:40:51,810 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629702_1099518554332] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 1038 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021682, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:40:52,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629705_1099518554335] hdfs.DFSClient: Slow ReadProcessor read fields took 1137ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1458596, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:52,092 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629704_1099518554334] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 25404 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 613680, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:52,403 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=513584, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/4c5e420f57c1409a9bebf48919ab2bcd 2015-10-16 12:40:52,411 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/4c5e420f57c1409a9bebf48919ab2bcd as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4c5e420f57c1409a9bebf48919ab2bcd 2015-10-16 12:40:52,420 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4c5e420f57c1409a9bebf48919ab2bcd, entries=466310, sequenceid=513584, filesize=70.8 M 2015-10-16 12:40:52,421 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134293200, currentsize=23.22 MB/24352720 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 925ms, sequenceid=513584, compaction requested=true 2015-10-16 12:40:52,421 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:40:53,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629702_1099518554332] hdfs.DFSClient: Slow ReadProcessor read fields took 1097ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1331312, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:40:53,324 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629704_1099518554334] hdfs.DFSClient: Slow ReadProcessor read fields took 186ms (threshold=20ms); ack: seqno: 31398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 186192939, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:53,324 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 186ms (threshold=20ms) 2015-10-16 12:40:53,324 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 186ms (threshold=20ms) 2015-10-16 12:40:53,324 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 187ms (threshold=20ms) 2015-10-16 12:40:53,324 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 187ms (threshold=20ms) 2015-10-16 12:40:53,324 INFO [sync.0] wal.FSHLog: Slow sync cost: 186 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:53,324 INFO [sync.1] wal.FSHLog: Slow sync cost: 186 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:53,324 INFO [sync.2] wal.FSHLog: Slow sync cost: 187 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:53,324 INFO [sync.3] wal.FSHLog: Slow sync cost: 186 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:53,324 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 187ms (threshold=20ms) 2015-10-16 12:40:53,324 INFO [sync.4] wal.FSHLog: Slow sync cost: 186 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:53,338 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629705_1099518554335] hdfs.DFSClient: Slow ReadProcessor read fields took 1150ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1417937, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:53,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629704_1099518554334] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 31672 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 496385, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:53,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629705_1099518554335] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 2560 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794816, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:53,399 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:53,399 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:53,399 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:53,399 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:53,399 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:53,720 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629704_1099518554334] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 33456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 21354665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:53,721 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:53,721 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:53,721 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:40:53,721 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:53,721 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:40:54,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629704_1099518554334] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 36742 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 535397, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:54,293 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629702_1099518554332] hdfs.DFSClient: Slow ReadProcessor read fields took 1160ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1300039, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:40:54,300 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:40:54,300 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:40:54,300 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:40:54,300 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:40:54,300 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:40:54,621 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629705_1099518554335] hdfs.DFSClient: Slow ReadProcessor read fields took 1135ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 11546480, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:54,722 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:40:54,764 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:40:54,791 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024447427 with entries=78220, filesize=122.54 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024454723 2015-10-16 12:40:54,793 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024440289 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024440289 2015-10-16 12:40:54,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629707_1099518554337] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1215354, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:55,081 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629707_1099518554337] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 1443 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 468292, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:55,082 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:55,082 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:55,082 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:40:55,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629702_1099518554332] hdfs.DFSClient: Slow ReadProcessor read fields took 1150ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1547610, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:40:55,883 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629705_1099518554335] hdfs.DFSClient: Slow ReadProcessor read fields took 1143ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1511552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:55,928 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629707_1099518554337] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 6217 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1105979, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:55,928 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629705_1099518554335] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 2872 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 763220, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:55,929 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:55,929 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:55,929 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:55,929 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:55,929 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:56,071 DEBUG [PriorityRpcServer.handler=7,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:40:56,071 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134273808 > lower bound=16777216 2015-10-16 12:40:56,072 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:40:56,095 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629707_1099518554337] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 6980 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 563214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:56,677 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:56,677 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:56,677 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629708_1099518554338] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 675 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995349, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:40:56,677 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629707_1099518554337] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 9834 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 584608, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:56,678 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:56,678 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:56,678 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:56,841 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629702_1099518554332] hdfs.DFSClient: Slow ReadProcessor read fields took 1163ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4914076, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:40:56,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=560219, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5c701c3de48640beac2a75dcca7c80b7 2015-10-16 12:40:56,989 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5c701c3de48640beac2a75dcca7c80b7 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5c701c3de48640beac2a75dcca7c80b7 2015-10-16 12:40:57,003 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5c701c3de48640beac2a75dcca7c80b7, entries=466310, sequenceid=560219, filesize=70.8 M 2015-10-16 12:40:57,005 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134293840, currentsize=24.18 MB/25357840 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 933ms, sequenceid=560219, compaction requested=true 2015-10-16 12:40:57,006 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 12:40:57,154 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629705_1099518554335] hdfs.DFSClient: Slow ReadProcessor read fields took 1124ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1450625, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:57,355 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:57,355 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:57,355 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:57,355 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:57,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629707_1099518554337] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 13129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 457811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:57,355 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:58,110 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629702_1099518554332] hdfs.DFSClient: Slow ReadProcessor read fields took 1157ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1200993, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:40:58,267 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:58,267 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629707_1099518554337] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 18013 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 453836, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:58,267 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629702_1099518554332] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 1997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 754830, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:40:58,267 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:58,267 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:58,267 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:58,427 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629705_1099518554335] hdfs.DFSClient: Slow ReadProcessor read fields took 1147ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1500067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:58,721 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629707_1099518554337] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 20217 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 496045, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:58,721 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:40:58,721 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:40:58,721 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:40:58,721 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:40:58,721 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:40:59,218 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629707_1099518554337] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 22936 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 509482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:59,219 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:59,219 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:59,219 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:59,219 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:59,219 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:40:59,382 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629702_1099518554332] hdfs.DFSClient: Slow ReadProcessor read fields took 1111ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1484606, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:40:59,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629707_1099518554337] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 24361 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 590158, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:59,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629705_1099518554335] hdfs.DFSClient: Slow ReadProcessor read fields took 1168ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 14640213, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:40:59,780 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#1 average throughput is 7.26 MB/sec, slept 20 time(s) and total slept time is 25602 ms. 1 active compactions remaining, total limit is 15.71 MB/sec 2015-10-16 12:40:59,813 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/2ab4a2928d1d4cdda3dacc21149f9a5c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2ab4a2928d1d4cdda3dacc21149f9a5c 2015-10-16 12:40:59,830 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:40:59,833 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:40:59,840 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/16c7103ecf9849228b4a4ee8e4f36e3b, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/16c7103ecf9849228b4a4ee8e4f36e3b 2015-10-16 12:40:59,848 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9fd933f5ae4f46ddbaa0016a0c5d73cc, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9fd933f5ae4f46ddbaa0016a0c5d73cc 2015-10-16 12:40:59,856 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cdd51822d73740e9b9ddbe442388e907, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cdd51822d73740e9b9ddbe442388e907 2015-10-16 12:40:59,857 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 2ab4a2928d1d4cdda3dacc21149f9a5c(size=212.5 M), total size for store is 850.4 M. This selection was in queue for 0sec, and took 28sec to execute. 2015-10-16 12:40:59,857 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=4, time=36902340141001204; duration=28sec 2015-10-16 12:40:59,857 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 12:40:59,857 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 3 compacting, 3 eligible, 10 blocking 2015-10-16 12:40:59,859 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222850121 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 12:40:59,859 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:40:59,859 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:40:59,859 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 12:40:59,860 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/dbe4b98577ce49f4853db95776593add, keycount=466330, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=466949 2015-10-16 12:40:59,860 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4c5e420f57c1409a9bebf48919ab2bcd, keycount=466310, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=513584 2015-10-16 12:40:59,860 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5c701c3de48640beac2a75dcca7c80b7, keycount=466310, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=560219 2015-10-16 12:40:59,879 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:40:59,970 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629707_1099518554337] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 26903 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 542916, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:40:59,970 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629710_1099518554340] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 64 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4115511, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:40:59,970 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:59,970 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:59,970 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:40:59,970 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:40:59,970 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:00,562 DEBUG [PriorityRpcServer.handler=12,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:41:00,562 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134291856 > lower bound=16777216 2015-10-16 12:41:00,562 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:41:00,589 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629707_1099518554337] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 30396 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 488430, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:00,653 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629709_1099518554339] hdfs.DFSClient: Slow ReadProcessor read fields took 1139ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1241677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:00,763 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629711_1099518554341] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 789657, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:00,763 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629709_1099518554339] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 2280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912101, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:00,763 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629707_1099518554337] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 31209 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 426029, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:00,764 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:00,764 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:00,764 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:00,764 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:01,165 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629710_1099518554340] hdfs.DFSClient: Slow ReadProcessor read fields took 1131ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2138797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:41:01,348 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629711_1099518554341] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1096 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794655, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:01,348 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629707_1099518554337] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 34154 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 567373, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:01,349 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:01,349 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:01,349 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:01,349 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:01,390 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=606856, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/3a8c5d5eb9894288aa0de174ccc93477 2015-10-16 12:41:01,396 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/3a8c5d5eb9894288aa0de174ccc93477 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3a8c5d5eb9894288aa0de174ccc93477 2015-10-16 12:41:01,411 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3a8c5d5eb9894288aa0de174ccc93477, entries=466320, sequenceid=606856, filesize=70.8 M 2015-10-16 12:41:01,416 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134297200, currentsize=22.51 MB/23603200 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 854ms, sequenceid=606856, compaction requested=true 2015-10-16 12:41:01,417 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 12:41:01,925 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629709_1099518554339] hdfs.DFSClient: Slow ReadProcessor read fields took 1115ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1271839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:02,084 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629707_1099518554337] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 37713 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 760296, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:02,324 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:41:02,374 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024454723 with entries=77999, filesize=122.19 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024462324 2015-10-16 12:41:02,375 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024447427 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024447427 2015-10-16 12:41:02,376 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629712_1099518554342] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:41:02,437 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629710_1099518554340] hdfs.DFSClient: Slow ReadProcessor read fields took 1148ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1348961, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:41:03,188 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629712_1099518554342] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 4604 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 377972, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:41:03,189 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:03,189 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:03,189 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:03,189 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:41:03,197 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629709_1099518554339] hdfs.DFSClient: Slow ReadProcessor read fields took 1155ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1316469, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:03,768 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629710_1099518554340] hdfs.DFSClient: Slow ReadProcessor read fields took 1214ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 61156495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:41:03,909 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629712_1099518554342] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 8658 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 425801, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:41:03,909 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:41:03,909 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:41:03,909 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:41:03,909 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:41:03,909 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:41:04,496 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629712_1099518554342] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 11721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 411302, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:41:04,498 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629709_1099518554339] hdfs.DFSClient: Slow ReadProcessor read fields took 1176ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1049730, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:04,515 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 92ms (threshold=20ms) 2015-10-16 12:41:04,515 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 92ms (threshold=20ms) 2015-10-16 12:41:04,515 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 92ms (threshold=20ms) 2015-10-16 12:41:04,515 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 91ms (threshold=20ms) 2015-10-16 12:41:04,515 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 91ms (threshold=20ms) 2015-10-16 12:41:04,862 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629712_1099518554342] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 13530 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 659606, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:41:04,862 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:04,862 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:04,862 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:04,862 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:04,980 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629710_1099518554340] hdfs.DFSClient: Slow ReadProcessor read fields took 1147ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1321959, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:41:05,056 DEBUG [PriorityRpcServer.handler=6,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:41:05,056 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134280048 > lower bound=16777216 2015-10-16 12:41:05,057 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:41:05,081 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629712_1099518554342] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 14618 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 641724, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:41:05,594 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629713_1099518554343] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 582 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779921, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:05,594 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629712_1099518554342] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 17286 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 413305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:41:05,594 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:05,594 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:05,594 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:05,594 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:05,594 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:05,770 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629709_1099518554339] hdfs.DFSClient: Slow ReadProcessor read fields took 1154ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1105747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:05,990 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=653494, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/10d4311b0aff40c79834bde183d38790 2015-10-16 12:41:05,997 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/10d4311b0aff40c79834bde183d38790 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/10d4311b0aff40c79834bde183d38790 2015-10-16 12:41:06,010 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/10d4311b0aff40c79834bde183d38790, entries=466340, sequenceid=653494, filesize=70.8 M 2015-10-16 12:41:06,012 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134301520, currentsize=27.32 MB/28648640 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 956ms, sequenceid=653494, compaction requested=true 2015-10-16 12:41:06,013 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:3), split_queue=0, merge_queue=0 2015-10-16 12:41:06,268 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:41:06,270 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629710_1099518554340] hdfs.DFSClient: Slow ReadProcessor read fields took 1167ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1157139, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:41:07,042 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629709_1099518554339] hdfs.DFSClient: Slow ReadProcessor read fields took 1150ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1189362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:07,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629709_1099518554339] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 3073 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 664882, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:07,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629712_1099518554342] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 25637 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 640835, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:41:07,114 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:07,114 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:07,114 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:07,114 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:07,114 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:07,542 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629710_1099518554340] hdfs.DFSClient: Slow ReadProcessor read fields took 1155ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1130434, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:41:07,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629712_1099518554342] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 30117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 564870, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:41:07,942 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:07,942 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:07,942 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:07,942 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:07,942 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:08,314 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629709_1099518554339] hdfs.DFSClient: Slow ReadProcessor read fields took 1129ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1107271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:08,819 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629712_1099518554342] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 35039 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 542834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:41:08,819 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:08,819 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:08,819 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:08,820 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:08,821 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629710_1099518554340] hdfs.DFSClient: Slow ReadProcessor read fields took 1163ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1402324, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:41:09,352 DEBUG [PriorityRpcServer.handler=12,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:41:09,352 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134264736 > lower bound=16777216 2015-10-16 12:41:09,352 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:41:09,378 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629712_1099518554342] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 38111 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 474036, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:41:09,549 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:41:09,586 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629709_1099518554339] hdfs.DFSClient: Slow ReadProcessor read fields took 1139ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1063526, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:09,590 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024462324 with entries=77922, filesize=122.08 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024469549 2015-10-16 12:41:09,592 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024454723 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024454723 2015-10-16 12:41:09,593 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629715_1099518554345] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1096820, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:09,618 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629714_1099518554344] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 305 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 826059, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:09,618 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629715_1099518554345] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 4 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 647787, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:09,618 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629709_1099518554339] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 3356 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066757, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:09,619 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:09,619 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:09,619 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:09,619 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:09,619 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:09,729 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#2 average throughput is 8.08 MB/sec, slept 20 time(s) and total slept time is 22730 ms. 1 active compactions remaining, total limit is 15.71 MB/sec 2015-10-16 12:41:09,750 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b4c8a0fc21be4114b78df88d24074046 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b4c8a0fc21be4114b78df88d24074046 2015-10-16 12:41:09,762 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:41:09,766 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:41:09,773 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c2e11276b1544ef580a311f1987f4d3b, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c2e11276b1544ef580a311f1987f4d3b 2015-10-16 12:41:09,781 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b1993562eff143109b9a4148f892a2ba, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b1993562eff143109b9a4148f892a2ba 2015-10-16 12:41:09,789 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6a3b7e8d9b114a74ba2f7ce53df7b9aa, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6a3b7e8d9b114a74ba2f7ce53df7b9aa 2015-10-16 12:41:09,790 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into b4c8a0fc21be4114b78df88d24074046(size=213.1 M), total size for store is 992.1 M. This selection was in queue for 0sec, and took 25sec to execute. 2015-10-16 12:41:09,790 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=213.1 M, priority=3, time=36902352928802604; duration=25sec 2015-10-16 12:41:09,790 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:3), split_queue=0, merge_queue=0 2015-10-16 12:41:09,790 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 3 compacting, 2 eligible, 10 blocking 2015-10-16 12:41:09,791 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:41:09,791 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:41:09,791 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:41:09,791 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 3 compacting, 2 eligible, 10 blocking 2015-10-16 12:41:09,792 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:41:09,792 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:41:09,792 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:41:09,793 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 3 compacting, 2 eligible, 10 blocking 2015-10-16 12:41:09,794 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:41:09,794 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:41:09,794 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:41:10,093 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629710_1099518554340] hdfs.DFSClient: Slow ReadProcessor read fields took 1152ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1219504, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:41:10,203 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629710_1099518554340] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1430 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794370, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:41:10,203 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629714_1099518554344] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1120 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 693898, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:10,203 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:10,203 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629715_1099518554345] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 3129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 490192, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:10,203 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:10,203 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:10,203 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:10,203 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:10,248 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=700121, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/93ca39cacd4f4416ac0820cd055efe7c 2015-10-16 12:41:10,255 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/93ca39cacd4f4416ac0820cd055efe7c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/93ca39cacd4f4416ac0820cd055efe7c 2015-10-16 12:41:10,269 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/93ca39cacd4f4416ac0820cd055efe7c, entries=466230, sequenceid=700121, filesize=70.8 M 2015-10-16 12:41:10,270 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134270080, currentsize=25.39 MB/26627680 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 918ms, sequenceid=700121, compaction requested=true 2015-10-16 12:41:10,270 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:41:10,271 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 3 compacting, 3 eligible, 10 blocking 2015-10-16 12:41:10,272 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222840786 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 12:41:10,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:41:10,273 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:41:10,273 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 12:41:10,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3a8c5d5eb9894288aa0de174ccc93477, keycount=466320, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=606856 2015-10-16 12:41:10,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/10d4311b0aff40c79834bde183d38790, keycount=466340, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=653494 2015-10-16 12:41:10,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/93ca39cacd4f4416ac0820cd055efe7c, keycount=466230, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=700121 2015-10-16 12:41:10,293 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:41:10,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629710_1099518554340] hdfs.DFSClient: Slow ReadProcessor read fields took 473ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1332265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:41:10,903 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629715_1099518554345] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 6988 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 405501, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:10,903 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:10,903 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:10,903 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:10,904 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:10,904 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:11,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629715_1099518554345] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 9514 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 634360, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:11,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629715_1099518554345] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 10086 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 435740, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:11,608 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 82ms (threshold=20ms) 2015-10-16 12:41:11,608 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 82ms (threshold=20ms) 2015-10-16 12:41:11,608 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 82ms (threshold=20ms) 2015-10-16 12:41:11,609 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 82ms (threshold=20ms) 2015-10-16 12:41:11,609 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 82ms (threshold=20ms) 2015-10-16 12:41:11,609 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 1161ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1272881, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:11,950 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629715_1099518554345] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 11895 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 540175, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:11,950 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:11,950 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:11,950 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:11,950 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:11,950 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:12,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629710_1099518554340] hdfs.DFSClient: Slow ReadProcessor read fields took 1150ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1346660, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:41:12,761 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629715_1099518554345] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 16752 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 506161, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:12,761 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:12,761 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:12,761 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:12,761 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:12,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 1128ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1098417, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:13,274 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629710_1099518554340] hdfs.DFSClient: Slow ReadProcessor read fields took 1135ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1455999, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:41:13,553 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629715_1099518554345] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 20974 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 574050, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:13,554 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:13,554 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:13,554 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:13,554 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:13,554 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:13,830 DEBUG [PriorityRpcServer.handler=19,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:41:13,830 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134271056 > lower bound=16777216 2015-10-16 12:41:13,830 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:41:13,854 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629715_1099518554345] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 22552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 628084, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:14,153 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 1132ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1163588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:14,307 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 664 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 768706, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:14,307 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629717_1099518554347] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 569 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 952017, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:14,307 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629715_1099518554345] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 24741 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 499886, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:14,307 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:14,307 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:14,307 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:14,307 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:14,307 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:14,547 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629710_1099518554340] hdfs.DFSClient: Slow ReadProcessor read fields took 1150ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1340151, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:41:14,739 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=746758, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d0d691e5fe5e49b1902d3974b458167e 2015-10-16 12:41:14,746 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d0d691e5fe5e49b1902d3974b458167e as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d0d691e5fe5e49b1902d3974b458167e 2015-10-16 12:41:14,760 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d0d691e5fe5e49b1902d3974b458167e, entries=466320, sequenceid=746758, filesize=70.8 M 2015-10-16 12:41:14,762 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134296560, currentsize=25.68 MB/26924560 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 932ms, sequenceid=746758, compaction requested=true 2015-10-16 12:41:14,762 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:41:15,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629715_1099518554345] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 28506 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 625607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:15,018 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:15,018 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:15,018 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:15,018 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:15,018 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:15,425 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 1118ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1248292, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:15,467 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629715_1099518554345] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 30876 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 690724, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:15,818 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629718_1099518554348] hdfs.DFSClient: Slow ReadProcessor read fields took 1149ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1140952, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:41:15,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629718_1099518554348] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 2225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 711083, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:41:15,880 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629715_1099518554345] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 33095 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 372285, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:15,880 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:15,880 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:15,880 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:15,880 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:15,880 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:16,697 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 1153ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 953755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:16,786 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629715_1099518554345] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 37923 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 572875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:16,786 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 917 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 761769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:16,786 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:16,786 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:16,786 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:16,786 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:16,786 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:17,003 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:41:17,056 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024469549 with entries=77975, filesize=122.16 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024477003 2015-10-16 12:41:17,058 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024462324 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024462324 2015-10-16 12:41:17,060 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629719_1099518554349] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1057746, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:41:17,090 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629718_1099518554348] hdfs.DFSClient: Slow ReadProcessor read fields took 1127ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1091096, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:41:17,582 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629719_1099518554349] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 3066 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 446128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:41:17,582 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:17,582 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:17,582 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:17,582 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:17,582 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:17,979 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 1132ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 11660162, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:18,180 DEBUG [PriorityRpcServer.handler=13,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:41:18,180 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134265136 > lower bound=16777216 2015-10-16 12:41:18,180 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:41:18,207 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629719_1099518554349] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 6636 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 481935, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:41:18,380 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629719_1099518554349] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 7509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 443514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:41:18,380 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:18,380 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:18,380 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:18,380 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629720_1099518554350] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 209 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 720689, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:41:18,380 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:18,380 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:18,382 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629718_1099518554348] hdfs.DFSClient: Slow ReadProcessor read fields took 1175ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1140272, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:41:19,002 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:41:19,002 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:41:19,089 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=793389, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5f7aee38d3694b4a9448e00655b77de4 2015-10-16 12:41:19,096 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5f7aee38d3694b4a9448e00655b77de4 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5f7aee38d3694b4a9448e00655b77de4 2015-10-16 12:41:19,110 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5f7aee38d3694b4a9448e00655b77de4, entries=466270, sequenceid=793389, filesize=70.8 M 2015-10-16 12:41:19,112 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134282000, currentsize=26.20 MB/27477040 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 932ms, sequenceid=793389, compaction requested=true 2015-10-16 12:41:19,112 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 12:41:19,241 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 1132ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:19,654 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629718_1099518554348] hdfs.DFSClient: Slow ReadProcessor read fields took 1127ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1018419, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:41:19,760 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629718_1099518554348] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 2773 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 756584, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:41:19,760 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629719_1099518554349] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 15008 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 396503, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:41:19,760 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:19,760 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:19,760 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:19,760 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:19,760 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:20,349 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629719_1099518554349] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 18124 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 635309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:41:20,514 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 1134ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:20,652 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629719_1099518554349] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 19726 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 379931, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:41:20,652 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:20,652 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:20,652 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:20,652 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:20,652 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:20,653 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 1477 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869052, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:20,926 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629718_1099518554348] hdfs.DFSClient: Slow ReadProcessor read fields took 1116ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1087079, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:41:21,473 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629719_1099518554349] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 24526 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 381470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:41:21,474 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:41:21,474 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:21,474 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:21,474 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:41:21,474 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:21,785 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 1113ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 991021, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:22,080 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629719_1099518554349] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 27590 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 430449, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:41:22,080 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 73ms (threshold=20ms) 2015-10-16 12:41:22,080 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 73ms (threshold=20ms) 2015-10-16 12:41:22,080 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 73ms (threshold=20ms) 2015-10-16 12:41:22,080 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 73ms (threshold=20ms) 2015-10-16 12:41:22,080 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 73ms (threshold=20ms) 2015-10-16 12:41:22,198 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629718_1099518554348] hdfs.DFSClient: Slow ReadProcessor read fields took 1150ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022671, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:41:22,321 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629718_1099518554348] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 3138 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 816982, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:41:22,321 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629719_1099518554349] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 28918 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912767, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:41:22,321 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:22,322 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:22,322 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:22,321 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:22,322 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:22,559 DEBUG [PriorityRpcServer.handler=2,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:41:22,559 WARN [MemStoreFlusher.0] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:41:22,559 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:3), split_queue=0, merge_queue=0 2015-10-16 12:41:23,057 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 1141ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1104794, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:23,060 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629719_1099518554349] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 32927 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 50190203, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:41:23,060 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:41:23,060 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:41:23,060 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 12:41:23,060 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:41:23,060 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:41:23,207 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 1819 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844389, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:23,207 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629719_1099518554349] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 33634 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 467683, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:41:23,208 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:23,208 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:23,208 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:23,208 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:23,470 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629718_1099518554348] hdfs.DFSClient: Slow ReadProcessor read fields took 1119ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1035734, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:41:24,040 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629719_1099518554349] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 38495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 395479, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:41:24,041 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:24,041 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:24,041 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:24,041 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:24,041 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:24,179 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:41:24,231 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024477003 with entries=77996, filesize=122.19 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024484179 2015-10-16 12:41:24,232 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024469549 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024469549 2015-10-16 12:41:24,234 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629721_1099518554351] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1040035, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:24,329 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 1111ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1083407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:24,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629718_1099518554348] hdfs.DFSClient: Slow ReadProcessor read fields took 1151ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1380003, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:41:24,823 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#3 average throughput is 8.29 MB/sec, slept 20 time(s) and total slept time is 22164 ms. 1 active compactions remaining, total limit is 15.71 MB/sec 2015-10-16 12:41:24,844 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/0b7364a0f78146b9baca89016d5c720b as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0b7364a0f78146b9baca89016d5c720b 2015-10-16 12:41:24,852 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:41:24,855 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:41:24,863 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/dbe4b98577ce49f4853db95776593add, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/dbe4b98577ce49f4853db95776593add 2015-10-16 12:41:24,894 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4c5e420f57c1409a9bebf48919ab2bcd, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4c5e420f57c1409a9bebf48919ab2bcd 2015-10-16 12:41:24,894 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629721_1099518554351] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 3441 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 603349, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:24,894 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:41:24,894 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:41:24,894 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:41:24,894 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:41:24,895 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:24,897 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5c701c3de48640beac2a75dcca7c80b7, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5c701c3de48640beac2a75dcca7c80b7 2015-10-16 12:41:24,897 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 0b7364a0f78146b9baca89016d5c720b(size=212.5 M), total size for store is 1.2 G. This selection was in queue for 0sec, and took 25sec to execute. 2015-10-16 12:41:24,897 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=2, time=36902368713183624; duration=25sec 2015-10-16 12:41:24,897 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:3), split_queue=0, merge_queue=0 2015-10-16 12:41:24,897 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 3 compacting, 2 eligible, 10 blocking 2015-10-16 12:41:24,898 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:41:24,898 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:41:24,898 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:41:24,898 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 3 compacting, 2 eligible, 10 blocking 2015-10-16 12:41:24,899 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:41:24,899 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:41:24,899 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:41:24,899 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 3 compacting, 2 eligible, 10 blocking 2015-10-16 12:41:24,900 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:41:24,900 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:41:24,900 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:41:25,260 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=218523536 > lower bound=16777216 2015-10-16 12:41:25,260 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=208.36 MB 2015-10-16 12:41:25,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629721_1099518554351] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 5594 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 614395, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:25,386 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629721_1099518554351] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 5820 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 543123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:25,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629716_1099518554346] hdfs.DFSClient: Slow ReadProcessor read fields took 1151ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044365, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:25,729 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629721_1099518554351] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 7509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 596307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:25,729 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629722_1099518554352] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912854, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:25,729 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:41:25,729 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629723_1099518554353] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 2090 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4122563, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:25,730 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:41:25,730 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:41:25,730 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:41:25,730 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:41:26,237 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629723_1099518554353] hdfs.DFSClient: Slow ReadProcessor read fields took 456ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1211379, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:26,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629723_1099518554353] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 2274 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 711447, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:26,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629722_1099518554352] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1313 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 774917, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:26,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629721_1099518554351] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 10879 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 394617, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:26,360 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:26,360 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:26,788 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=869273, memsize=208.4 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/7dbfe79cd8224dc3bfddb9836e94446c 2015-10-16 12:41:26,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/7dbfe79cd8224dc3bfddb9836e94446c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7dbfe79cd8224dc3bfddb9836e94446c 2015-10-16 12:41:26,810 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7dbfe79cd8224dc3bfddb9836e94446c, entries=758790, sequenceid=869273, filesize=115.3 M 2015-10-16 12:41:26,812 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~208.40 MB/218526000, currentsize=42.93 MB/45012400 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1551ms, sequenceid=869273, compaction requested=true 2015-10-16 12:41:26,812 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:41:26,812 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 3 compacting, 3 eligible, 10 blocking 2015-10-16 12:41:26,813 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 269455464 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 12:41:26,813 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:41:26,814 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:41:26,814 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=257.0 M 2015-10-16 12:41:26,814 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d0d691e5fe5e49b1902d3974b458167e, keycount=466320, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=746758 2015-10-16 12:41:26,814 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5f7aee38d3694b4a9448e00655b77de4, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=793389 2015-10-16 12:41:26,814 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7dbfe79cd8224dc3bfddb9836e94446c, keycount=758790, bloomtype=ROW, size=115.3 M, encoding=NONE, seqNum=869273 2015-10-16 12:41:26,832 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:41:26,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629723_1099518554353] hdfs.DFSClient: Slow ReadProcessor read fields took 457ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1045007, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:26,966 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:26,966 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629723_1099518554353] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 2433 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 831708, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:26,966 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 128 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 743800, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:26,966 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629721_1099518554351] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 14038 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 727792, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:26,967 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:26,967 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:26,967 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:26,967 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:27,476 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629721_1099518554351] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 16679 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 703678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:27,910 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629721_1099518554351] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 19175 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 401306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:27,910 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:27,910 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:27,910 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:27,910 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:27,910 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:28,113 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 1100ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983447, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:28,145 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629723_1099518554353] hdfs.DFSClient: Slow ReadProcessor read fields took 1120ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1362939, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:28,666 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629721_1099518554351] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 23344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 497398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:28,666 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:28,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629721_1099518554351] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 24054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838739, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:29,385 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 1131ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102518, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:29,435 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629723_1099518554353] hdfs.DFSClient: Slow ReadProcessor read fields took 1147ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 19371169, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:29,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629721_1099518554351] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 27842 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 665737, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:29,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629723_1099518554353] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 2790 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838053, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:29,534 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:29,534 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:29,534 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:29,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 487 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1094385, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:29,534 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:29,534 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:29,753 DEBUG [PriorityRpcServer.handler=4,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:41:29,753 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134270176 > lower bound=16777216 2015-10-16 12:41:29,813 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=129.86 MB 2015-10-16 12:41:29,838 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629721_1099518554351] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 29414 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 515824, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:30,315 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629725_1099518554355] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 704 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:30,315 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629721_1099518554351] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 31758 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 6105368, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:30,316 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:41:30,316 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:41:30,316 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:30,622 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=916584, memsize=129.9 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/cb6aec0ff68246e08bc60d42459da2a8 2015-10-16 12:41:30,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/cb6aec0ff68246e08bc60d42459da2a8 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cb6aec0ff68246e08bc60d42459da2a8 2015-10-16 12:41:30,643 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cb6aec0ff68246e08bc60d42459da2a8, entries=473070, sequenceid=916584, filesize=71.9 M 2015-10-16 12:41:30,645 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~129.93 MB/136239680, currentsize=22.41 MB/23503120 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 832ms, sequenceid=916584, compaction requested=true 2015-10-16 12:41:30,646 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:41:30,657 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 1114ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1192863, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:30,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629723_1099518554353] hdfs.DFSClient: Slow ReadProcessor read fields took 1115ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1177794, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:30,972 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629721_1099518554351] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 35045 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 580408, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:30,972 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:30,972 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:30,972 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:30,972 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:30,972 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:31,713 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:41:31,755 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024484179 with entries=77970, filesize=122.15 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024491713 2015-10-16 12:41:31,757 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024477003 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024477003 2015-10-16 12:41:31,758 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629726_1099518554356] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 981561, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:31,929 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 1160ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1324826, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:31,984 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629726_1099518554356] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1147 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069983, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:31,984 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 708 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 854761, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:31,984 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629723_1099518554353] hdfs.DFSClient: Slow ReadProcessor read fields took 1171ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1526105, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:31,984 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:31,984 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:31,984 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:31,984 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:31,984 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:32,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629726_1099518554356] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 5557 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 470105, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:32,784 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:32,784 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:32,784 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:32,784 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:32,784 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:33,218 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 1146ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 17981413, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:33,234 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629723_1099518554353] hdfs.DFSClient: Slow ReadProcessor read fields took 1124ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1344870, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:33,535 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629726_1099518554356] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 9797 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 423150, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:33,535 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:33,535 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:33,535 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:33,535 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:33,535 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:34,164 DEBUG [PriorityRpcServer.handler=19,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:41:34,164 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:41:34,165 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 12:41:34,473 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 1154ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1378087, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:34,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629723_1099518554353] hdfs.DFSClient: Slow ReadProcessor read fields took 1570ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 427411800, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:34,954 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#4 average throughput is 8.38 MB/sec, slept 20 time(s) and total slept time is 21222 ms. 1 active compactions remaining, total limit is 15.71 MB/sec 2015-10-16 12:41:34,996 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/0261c61815d0485497e8ee387c903d48 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0261c61815d0485497e8ee387c903d48 2015-10-16 12:41:34,999 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629726_1099518554356] hdfs.DFSClient: Slow ReadProcessor read fields took 796ms (threshold=20ms); ack: seqno: 13827 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 795998917, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:34,999 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 796ms (threshold=20ms) 2015-10-16 12:41:34,999 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 796ms (threshold=20ms) 2015-10-16 12:41:35,000 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 796ms (threshold=20ms) 2015-10-16 12:41:34,999 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 796ms (threshold=20ms) 2015-10-16 12:41:35,000 INFO [sync.3] wal.FSHLog: Slow sync cost: 796 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:35,000 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 796ms (threshold=20ms) 2015-10-16 12:41:35,000 INFO [sync.0] wal.FSHLog: Slow sync cost: 796 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:35,000 INFO [sync.2] wal.FSHLog: Slow sync cost: 796 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:35,000 INFO [sync.4] wal.FSHLog: Slow sync cost: 796 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:35,000 INFO [sync.1] wal.FSHLog: Slow sync cost: 796 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:35,019 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:41:35,022 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:41:35,029 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3a8c5d5eb9894288aa0de174ccc93477, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3a8c5d5eb9894288aa0de174ccc93477 2015-10-16 12:41:35,038 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/10d4311b0aff40c79834bde183d38790, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/10d4311b0aff40c79834bde183d38790 2015-10-16 12:41:35,046 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/93ca39cacd4f4416ac0820cd055efe7c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/93ca39cacd4f4416ac0820cd055efe7c 2015-10-16 12:41:35,046 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 0261c61815d0485497e8ee387c903d48(size=212.5 M), total size for store is 1.4 G. This selection was in queue for 0sec, and took 24sec to execute. 2015-10-16 12:41:35,047 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=2, time=36902379126627858; duration=24sec 2015-10-16 12:41:35,047 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 12:41:35,047 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 3 compacting, 1 eligible, 10 blocking 2015-10-16 12:41:35,047 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:41:35,047 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:41:35,048 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:41:35,048 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 3 compacting, 1 eligible, 10 blocking 2015-10-16 12:41:35,048 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:41:35,048 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:41:35,048 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:41:35,065 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=137452736 > lower bound=16777216 2015-10-16 12:41:35,065 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=131.05 MB 2015-10-16 12:41:35,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629726_1099518554356] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 14131 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 605990, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:35,747 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 1158ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2707852, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:35,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629726_1099518554356] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 17605 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 553725, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:35,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629727_1099518554357] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 977 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758639, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:35,800 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:35,800 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:35,800 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:35,800 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:35,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 1195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861758, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:35,800 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:35,897 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629726_1099518554356] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 17900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 641379, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:35,980 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=964320, memsize=131.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/278a2721631a450fb052b557170421e7 2015-10-16 12:41:35,986 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/278a2721631a450fb052b557170421e7 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/278a2721631a450fb052b557170421e7 2015-10-16 12:41:35,992 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/278a2721631a450fb052b557170421e7, entries=477310, sequenceid=964320, filesize=72.5 M 2015-10-16 12:41:35,994 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~131.09 MB/137460960, currentsize=24.00 MB/25165040 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 929ms, sequenceid=964320, compaction requested=true 2015-10-16 12:41:35,994 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:41:35,994 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 3 compacting, 2 eligible, 10 blocking 2015-10-16 12:41:35,995 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:41:35,995 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:41:35,995 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:41:36,382 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 439ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1468023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:36,553 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629726_1099518554356] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 21458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 524979, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:36,553 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:36,553 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:36,553 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:36,553 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:36,553 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:37,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 509ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1364722, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:37,440 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629726_1099518554356] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 26255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 444039, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:37,440 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:41:37,440 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:41:37,441 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:41:37,441 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:41:37,441 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:41:37,653 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 517ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1261076, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:37,866 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629726_1099518554356] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 28220 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 510180, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:38,290 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 514ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1397653, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:38,368 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629726_1099518554356] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 30898 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 539188, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:38,368 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 1899 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 764705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:38,369 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:41:38,369 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:41:38,369 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:41:38,369 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:41:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.0, tune compaction throughput to 20.00 MB/sec 2015-10-16 12:41:38,926 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629724_1099518554354] hdfs.DFSClient: Slow ReadProcessor read fields took 478ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1332587, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:39,187 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629726_1099518554356] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 35323 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 479913, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:39,187 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:41:39,187 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:41:39,187 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:41:39,187 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:41:39,187 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:41:39,425 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629728_1099518554358] hdfs.DFSClient: Slow ReadProcessor read fields took 375ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1483739, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:39,554 DEBUG [PriorityRpcServer.handler=14,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:41:39,554 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134267856 > lower bound=16777216 2015-10-16 12:41:39,555 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:41:39,651 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629726_1099518554356] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 37411 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 730844, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:39,925 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629728_1099518554358] hdfs.DFSClient: Slow ReadProcessor read fields took 376ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1430021, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:39,955 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:41:40,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629726_1099518554356] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 38960 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 478594, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:41:40,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629728_1099518554358] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 2407 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 724143, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:40,018 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:41:40,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629729_1099518554359] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 459 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 823211, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:41:40,018 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:41:40,018 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:41:40,018 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:41:40,018 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:41:40,019 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 56ms (threshold=20ms) 2015-10-16 12:41:40,064 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024491713 with entries=77871, filesize=121.99 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024499956 2015-10-16 12:41:40,066 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024484179 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024484179 2015-10-16 12:41:40,067 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629730_1099518554360] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1143842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:40,425 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629728_1099518554358] hdfs.DFSClient: Slow ReadProcessor read fields took 333ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089525, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:40,479 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1010948, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/f97a0f1a4a844224b0507a98c21392b7 2015-10-16 12:41:40,487 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/f97a0f1a4a844224b0507a98c21392b7 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f97a0f1a4a844224b0507a98c21392b7 2015-10-16 12:41:40,501 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f97a0f1a4a844224b0507a98c21392b7, entries=466240, sequenceid=1010948, filesize=70.8 M 2015-10-16 12:41:40,502 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134273200, currentsize=21.14 MB/22166560 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 947ms, sequenceid=1010948, compaction requested=true 2015-10-16 12:41:40,503 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:41:40,503 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 3 compacting, 3 eligible, 10 blocking 2015-10-16 12:41:40,504 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 225661670 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 12:41:40,504 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:41:40,504 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:41:40,504 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=215.2 M 2015-10-16 12:41:40,505 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cb6aec0ff68246e08bc60d42459da2a8, keycount=473070, bloomtype=ROW, size=71.9 M, encoding=NONE, seqNum=916584 2015-10-16 12:41:40,505 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/278a2721631a450fb052b557170421e7, keycount=477310, bloomtype=ROW, size=72.5 M, encoding=NONE, seqNum=964320 2015-10-16 12:41:40,505 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f97a0f1a4a844224b0507a98c21392b7, keycount=466240, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=1010948 2015-10-16 12:41:40,522 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:41:41,425 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629728_1099518554358] hdfs.DFSClient: Slow ReadProcessor read fields took 848ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1129883, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:41,528 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:41,528 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:41,528 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629728_1099518554358] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 2792 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 778579, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:41,528 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:41:41,528 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:41,529 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:41:41,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629731_1099518554361] hdfs.DFSClient: Slow ReadProcessor read fields took 833ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:42,426 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629728_1099518554358] hdfs.DFSClient: Slow ReadProcessor read fields took 853ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1308430, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:42,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629730_1099518554360] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 12081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 879397, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:42,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629728_1099518554358] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 2853 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945116, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:42,459 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:42,459 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:42,459 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:42,459 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:42,459 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:42,531 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629731_1099518554361] hdfs.DFSClient: Slow ReadProcessor read fields took 859ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1104694, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:43,255 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629730_1099518554360] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 16341 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 554582, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:43,255 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:41:43,255 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:41:43,255 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:41:43,255 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:41:43,256 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:41:43,426 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629728_1099518554358] hdfs.DFSClient: Slow ReadProcessor read fields took 855ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1116530, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:43,532 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629731_1099518554361] hdfs.DFSClient: Slow ReadProcessor read fields took 828ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069874, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:44,067 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629730_1099518554360] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 20539 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 453168, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:44,067 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:44,067 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:44,067 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:44,067 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:44,067 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:44,288 DEBUG [PriorityRpcServer.handler=2,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:41:44,288 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:41:44,288 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:41:44,426 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629728_1099518554358] hdfs.DFSClient: Slow ReadProcessor read fields took 876ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207405, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:44,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629731_1099518554361] hdfs.DFSClient: Slow ReadProcessor read fields took 878ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:44,853 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629730_1099518554360] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 24715 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 385328, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:44,853 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:41:44,853 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:41:44,853 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:41:44,853 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:41:44,853 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:41:45,426 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629728_1099518554358] hdfs.DFSClient: Slow ReadProcessor read fields took 881ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1452849, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:45,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629731_1099518554361] hdfs.DFSClient: Slow ReadProcessor read fields took 875ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1284254, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:45,734 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629730_1099518554360] hdfs.DFSClient: Slow ReadProcessor read fields took 754ms (threshold=20ms); ack: seqno: 25380 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 753609043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:45,735 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 754ms (threshold=20ms) 2015-10-16 12:41:45,735 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 755ms (threshold=20ms) 2015-10-16 12:41:45,735 INFO [sync.4] wal.FSHLog: Slow sync cost: 754 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:45,735 INFO [sync.0] wal.FSHLog: Slow sync cost: 754 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:45,735 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 755ms (threshold=20ms) 2015-10-16 12:41:45,735 INFO [sync.3] wal.FSHLog: Slow sync cost: 754 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:45,735 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 755ms (threshold=20ms) 2015-10-16 12:41:45,735 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 755ms (threshold=20ms) 2015-10-16 12:41:45,735 INFO [sync.1] wal.FSHLog: Slow sync cost: 754 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:45,735 INFO [sync.2] wal.FSHLog: Slow sync cost: 754 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:46,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629730_1099518554360] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 28906 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 593603, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:46,413 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:46,413 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:46,413 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:46,413 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:41:46,426 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629728_1099518554358] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1218802, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:46,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629731_1099518554361] hdfs.DFSClient: Slow ReadProcessor read fields took 899ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994052, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:46,843 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629730_1099518554360] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 30773 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 583495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:47,290 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629730_1099518554360] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 33082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 603514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:47,290 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:41:47,290 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:41:47,290 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:41:47,290 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:41:47,290 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:41:47,426 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629728_1099518554358] hdfs.DFSClient: Slow ReadProcessor read fields took 857ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1120590, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:47,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629731_1099518554361] hdfs.DFSClient: Slow ReadProcessor read fields took 876ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1144858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:47,814 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629730_1099518554360] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 35520 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 20422230, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:47,815 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 74ms (threshold=20ms) 2015-10-16 12:41:47,815 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:41:47,815 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:41:47,815 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 74ms (threshold=20ms) 2015-10-16 12:41:47,815 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:41:48,170 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629730_1099518554360] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 37324 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 598974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:41:48,170 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:48,170 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:48,170 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:48,170 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:48,170 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:48,427 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629728_1099518554358] hdfs.DFSClient: Slow ReadProcessor read fields took 853ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1245940, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:48,479 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:41:48,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629731_1099518554361] hdfs.DFSClient: Slow ReadProcessor read fields took 873ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1149162, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:48,538 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024499956 with entries=78054, filesize=122.28 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024508479 2015-10-16 12:41:48,542 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629732_1099518554362] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1119057, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:48,978 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629732_1099518554362] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 2301 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 498581, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:48,978 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:48,978 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:48,978 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:48,978 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:48,978 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:49,426 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629728_1099518554358] hdfs.DFSClient: Slow ReadProcessor read fields took 851ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1109375, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:41:49,535 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629731_1099518554361] hdfs.DFSClient: Slow ReadProcessor read fields took 878ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3700869, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:49,558 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#5 average throughput is 11.00 MB/sec, slept 24 time(s) and total slept time is 19222 ms. 1 active compactions remaining, total limit is 20.00 MB/sec 2015-10-16 12:41:49,590 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:41:49,602 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/47014a9030d5469e9c456de21f92c5c4 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/47014a9030d5469e9c456de21f92c5c4 2015-10-16 12:41:49,618 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:41:49,621 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:41:49,629 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d0d691e5fe5e49b1902d3974b458167e, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d0d691e5fe5e49b1902d3974b458167e 2015-10-16 12:41:49,637 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5f7aee38d3694b4a9448e00655b77de4, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5f7aee38d3694b4a9448e00655b77de4 2015-10-16 12:41:49,645 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7dbfe79cd8224dc3bfddb9836e94446c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7dbfe79cd8224dc3bfddb9836e94446c 2015-10-16 12:41:49,646 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 47014a9030d5469e9c456de21f92c5c4(size=257.0 M), total size for store is 1.5 G. This selection was in queue for 0sec, and took 22sec to execute. 2015-10-16 12:41:49,646 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=257.0 M, priority=0, time=36902395667602717; duration=22sec 2015-10-16 12:41:49,646 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:41:49,646 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 3 compacting, 0 eligible, 10 blocking 2015-10-16 12:41:49,646 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:41:49,646 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:41:49,646 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:41:49,689 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=269902976 > lower bound=16777216 2015-10-16 12:41:49,689 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=257.35 MB 2015-10-16 12:41:49,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629732_1099518554362] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 6318 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:49,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629732_1099518554362] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 6403 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 593627, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:49,788 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:49,788 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:49,788 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:49,788 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:49,788 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:50,034 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629731_1099518554361] hdfs.DFSClient: Slow ReadProcessor read fields took 372ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1139869, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:50,382 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629732_1099518554362] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 9679 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 447070, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:50,382 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629734_1099518554364] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 870 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:50,382 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:41:50,382 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:41:50,382 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:41:50,382 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:41:50,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629731_1099518554361] hdfs.DFSClient: Slow ReadProcessor read fields took 347ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1119146, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:50,614 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629731_1099518554361] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 1847 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 72893584, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:51,034 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629731_1099518554361] hdfs.DFSClient: Slow ReadProcessor read fields took 356ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874243, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:41:51,387 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1104675, memsize=257.4 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1bae3cb3341042cca6e270ad3b99890b 2015-10-16 12:41:51,394 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1bae3cb3341042cca6e270ad3b99890b as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1bae3cb3341042cca6e270ad3b99890b 2015-10-16 12:41:51,408 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1bae3cb3341042cca6e270ad3b99890b, entries=937220, sequenceid=1104675, filesize=142.4 M 2015-10-16 12:41:51,411 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~257.41 MB/269911200, currentsize=49.18 MB/51573440 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1722ms, sequenceid=1104675, compaction requested=true 2015-10-16 12:41:51,411 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:41:51,411 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 3 compacting, 1 eligible, 10 blocking 2015-10-16 12:41:51,412 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:41:51,412 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:41:51,412 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:41:51,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629735_1099518554365] hdfs.DFSClient: Slow ReadProcessor read fields took 346ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1275385, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:51,607 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629732_1099518554362] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 16306 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 477551, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:51,607 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:51,607 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:41:51,607 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:51,607 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:51,607 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:51,751 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629735_1099518554365] hdfs.DFSClient: Slow ReadProcessor read fields took 217ms (threshold=20ms); ack: seqno: 2173 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1424598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:52,033 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629735_1099518554365] hdfs.DFSClient: Slow ReadProcessor read fields took 219ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008887, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:52,351 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629732_1099518554362] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 20668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 540856, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:52,351 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:52,351 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:52,354 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:41:52,354 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:41:52,351 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:52,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629735_1099518554365] hdfs.DFSClient: Slow ReadProcessor read fields took 381ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1233044, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:52,790 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629732_1099518554362] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 22769 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 518929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:52,791 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 12:41:52,791 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 12:41:52,791 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 72ms (threshold=20ms) 2015-10-16 12:41:52,791 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 72ms (threshold=20ms) 2015-10-16 12:41:52,791 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 73ms (threshold=20ms) 2015-10-16 12:41:53,033 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629735_1099518554365] hdfs.DFSClient: Slow ReadProcessor read fields took 383ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1263182, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:53,183 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629732_1099518554362] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 24840 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 532413, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:53,183 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:53,183 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:53,183 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:53,183 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:53,184 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:53,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629735_1099518554365] hdfs.DFSClient: Slow ReadProcessor read fields took 375ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196907, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:54,026 DEBUG [PriorityRpcServer.handler=0,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:41:54,026 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134296576 > lower bound=16777216 2015-10-16 12:41:54,027 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.03 MB 2015-10-16 12:41:54,033 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629735_1099518554365] hdfs.DFSClient: Slow ReadProcessor read fields took 375ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1261155, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:54,058 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629735_1099518554365] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 3013 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089802, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:54,076 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629732_1099518554362] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 29697 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 578492, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:54,432 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629732_1099518554362] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 31204 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 691648, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:54,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629735_1099518554365] hdfs.DFSClient: Slow ReadProcessor read fields took 361ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 938918, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:54,665 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629732_1099518554362] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 32366 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 712373, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:54,665 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629737_1099518554367] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 785 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:41:54,666 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:54,665 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629735_1099518554365] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 3326 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 721178, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:54,666 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:54,666 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:54,666 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:41:54,666 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:41:54,920 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1151319, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5f90d9b1208e4e949b21247588f0a0e1 2015-10-16 12:41:54,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5f90d9b1208e4e949b21247588f0a0e1 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5f90d9b1208e4e949b21247588f0a0e1 2015-10-16 12:41:54,941 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5f90d9b1208e4e949b21247588f0a0e1, entries=466400, sequenceid=1151319, filesize=70.9 M 2015-10-16 12:41:54,943 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.10 MB/134319200, currentsize=23.32 MB/24449920 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 917ms, sequenceid=1151319, compaction requested=true 2015-10-16 12:41:54,943 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:41:54,943 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 3 compacting, 2 eligible, 10 blocking 2015-10-16 12:41:54,944 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:41:54,944 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:41:54,944 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:41:55,034 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629735_1099518554365] hdfs.DFSClient: Slow ReadProcessor read fields took 353ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062509, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:41:55,148 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#6 average throughput is 14.31 MB/sec, slept 20 time(s) and total slept time is 11645 ms. 0 active compactions remaining, total limit is 20.00 MB/sec 2015-10-16 12:41:55,169 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5a28df304f4c441b9194a01700d939cf as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5a28df304f4c441b9194a01700d939cf 2015-10-16 12:41:55,183 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:41:55,188 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:41:55,195 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cb6aec0ff68246e08bc60d42459da2a8, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cb6aec0ff68246e08bc60d42459da2a8 2015-10-16 12:41:55,203 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/278a2721631a450fb052b557170421e7, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/278a2721631a450fb052b557170421e7 2015-10-16 12:41:55,211 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f97a0f1a4a844224b0507a98c21392b7, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f97a0f1a4a844224b0507a98c21392b7 2015-10-16 12:41:55,212 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 5a28df304f4c441b9194a01700d939cf(size=215.2 M), total size for store is 1.7 G. This selection was in queue for 0sec, and took 14sec to execute. 2015-10-16 12:41:55,212 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=215.2 M, priority=-1, time=36902409358274741; duration=14sec 2015-10-16 12:41:55,212 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:41:55,466 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629732_1099518554362] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 36662 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 590999, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:55,832 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:41:56,134 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 297ms (threshold=20ms) 2015-10-16 12:41:56,137 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Slow sync cost: 299 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:56,145 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024508479 with entries=81362, filesize=127.46 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024515833 2015-10-16 12:41:56,148 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024491713 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024491713 2015-10-16 12:41:56,154 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024499956 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024499956 2015-10-16 12:41:56,352 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629738_1099518554368] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1153 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 361992, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:41:56,352 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:41:56,353 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:41:56,353 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:41:56,353 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:41:56,353 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:41:57,263 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:41:57,263 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:57,263 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:41:57,499 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629738_1099518554368] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 7993 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 643226, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:41:58,222 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629738_1099518554368] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 12317 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 393616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:41:58,222 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:41:58,222 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:41:58,222 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:41:58,222 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:41:58,222 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:41:58,227 DEBUG [PriorityRpcServer.handler=14,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:41:58,227 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134265904 > lower bound=16777216 2015-10-16 12:41:58,227 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:41:58,251 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629738_1099518554368] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 12366 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489083, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:41:58,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629739_1099518554369] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 865 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 743329, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:41:58,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629738_1099518554368] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 16163 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 507955, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:41:58,912 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:41:58,912 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:41:58,912 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:41:58,912 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:41:58,912 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:41:59,111 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1197952, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d8be11cac05a433db34a73b32597b3a4 2015-10-16 12:41:59,118 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d8be11cac05a433db34a73b32597b3a4 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d8be11cac05a433db34a73b32597b3a4 2015-10-16 12:41:59,132 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d8be11cac05a433db34a73b32597b3a4, entries=466280, sequenceid=1197952, filesize=70.8 M 2015-10-16 12:41:59,134 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134283920, currentsize=28.28 MB/29654000 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 907ms, sequenceid=1197952, compaction requested=true 2015-10-16 12:41:59,134 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:41:59,134 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 0 compacting, 10 eligible, 10 blocking 2015-10-16 12:41:59,139 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 297872988 starting at candidate #7 after considering 36 permutations with 36 in ratio 2015-10-16 12:41:59,139 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:41:59,139 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:41:59,139 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=284.1 M 2015-10-16 12:41:59,139 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1bae3cb3341042cca6e270ad3b99890b, keycount=937220, bloomtype=ROW, size=142.4 M, encoding=NONE, seqNum=1104675 2015-10-16 12:41:59,139 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5f90d9b1208e4e949b21247588f0a0e1, keycount=466400, bloomtype=ROW, size=70.9 M, encoding=NONE, seqNum=1151319 2015-10-16 12:41:59,139 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d8be11cac05a433db34a73b32597b3a4, keycount=466280, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=1197952 2015-10-16 12:41:59,159 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:41:59,630 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629738_1099518554368] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 20431 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 396587, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:41:59,630 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:41:59,630 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:41:59,672 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629740_1099518554370] hdfs.DFSClient: Slow ReadProcessor read fields took 365ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1330759, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:42:00,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629740_1099518554370] hdfs.DFSClient: Slow ReadProcessor read fields took 372ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1467400, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:42:00,385 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629738_1099518554368] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 24642 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 557077, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:00,385 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:00,385 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:00,385 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:00,385 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:00,385 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:00,672 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629740_1099518554370] hdfs.DFSClient: Slow ReadProcessor read fields took 375ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1443075, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:42:00,892 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629738_1099518554368] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 27179 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 664744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:01,172 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629740_1099518554370] hdfs.DFSClient: Slow ReadProcessor read fields took 361ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1339003, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:42:01,257 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629740_1099518554370] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 740 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827244, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:42:01,257 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629738_1099518554368] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 29245 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 457736, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:01,257 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:01,257 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:01,257 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:01,257 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:01,257 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:01,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629740_1099518554370] hdfs.DFSClient: Slow ReadProcessor read fields took 337ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1415005, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:42:02,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629738_1099518554368] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 33746 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 483908, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:02,030 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:02,030 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:02,030 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:02,030 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:02,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629740_1099518554370] hdfs.DFSClient: Slow ReadProcessor read fields took 356ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1476009, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:42:02,403 DEBUG [PriorityRpcServer.handler=1,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:42:02,403 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134268016 > lower bound=16777216 2015-10-16 12:42:02,403 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:42:02,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629738_1099518554368] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 35992 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 624819, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:02,672 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629740_1099518554370] hdfs.DFSClient: Slow ReadProcessor read fields took 370ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1284117, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:42:02,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629738_1099518554368] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 37602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 467499, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:02,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629741_1099518554371] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 373 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728626, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:42:02,742 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:02,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629740_1099518554370] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 1224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 713180, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:42:02,742 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:02,742 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:02,742 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:03,027 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:42:03,070 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024515833 with entries=78022, filesize=122.23 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024523027 2015-10-16 12:42:03,072 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024508479 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024508479 2015-10-16 12:42:03,172 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629740_1099518554370] hdfs.DFSClient: Slow ReadProcessor read fields took 345ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1251046, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:42:03,261 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1244581, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8beba6ad43e24a6fada0575aec4e0ded 2015-10-16 12:42:03,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8beba6ad43e24a6fada0575aec4e0ded as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8beba6ad43e24a6fada0575aec4e0ded 2015-10-16 12:42:03,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629740_1099518554370] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1489 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 723407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:42:03,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629742_1099518554372] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 447419, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:03,304 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:03,304 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:03,304 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:03,304 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:03,304 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:03,304 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8beba6ad43e24a6fada0575aec4e0ded, entries=466250, sequenceid=1244581, filesize=70.8 M 2015-10-16 12:42:03,306 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134276240, currentsize=25.36 MB/26593360 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 903ms, sequenceid=1244581, compaction requested=true 2015-10-16 12:42:03,306 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:42:03,306 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 3 compacting, 1 eligible, 10 blocking 2015-10-16 12:42:03,307 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:42:03,307 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:42:03,307 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:42:03,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629740_1099518554370] hdfs.DFSClient: Slow ReadProcessor read fields took 362ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1574873, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:42:04,144 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629742_1099518554372] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 5889 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 463867, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:04,145 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:04,145 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:04,145 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:04,145 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:04,145 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:04,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629740_1099518554370] hdfs.DFSClient: Slow ReadProcessor read fields took 385ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1033528, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:42:04,627 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629742_1099518554372] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 8211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 509274, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:04,627 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 73ms (threshold=20ms) 2015-10-16 12:42:04,627 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 74ms (threshold=20ms) 2015-10-16 12:42:04,627 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 73ms (threshold=20ms) 2015-10-16 12:42:04,627 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 73ms (threshold=20ms) 2015-10-16 12:42:04,627 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 73ms (threshold=20ms) 2015-10-16 12:42:04,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629740_1099518554370] hdfs.DFSClient: Slow ReadProcessor read fields took 382ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1040127, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:42:04,977 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629742_1099518554372] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 10003 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 434748, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:04,977 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:04,977 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:04,977 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:04,977 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:04,977 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:05,174 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629740_1099518554370] hdfs.DFSClient: Slow ReadProcessor read fields took 383ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1232492, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:42:05,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 377ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1064161, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:05,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629742_1099518554372] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 14456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 394104, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:05,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 2283 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790509, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:05,781 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:05,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629742_1099518554372] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 15099 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 544953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:05,912 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:05,912 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:05,912 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:05,912 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:05,912 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:06,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 351ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081396, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:06,600 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629742_1099518554372] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 19060 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 386184, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:06,601 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:06,601 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:06,601 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:06,601 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:06,601 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:06,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 377ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174403, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:06,757 DEBUG [PriorityRpcServer.handler=18,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:42:06,757 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:42:06,758 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:42:06,758 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 3 compacting, 1 eligible, 10 blocking 2015-10-16 12:42:06,759 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:42:06,759 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:42:06,759 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:42:07,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 382ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1120906, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:07,367 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:07,367 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629742_1099518554372] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 23255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 404168, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:07,367 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:07,367 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:07,566 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629742_1099518554372] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 23941 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 823360, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:07,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 380ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1100371, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:08,070 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:08,070 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:08,070 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:08,071 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:08,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 379ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1206735, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:08,259 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 3100 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904146, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:08,259 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629742_1099518554372] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 27736 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 532246, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:08,259 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:08,259 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:08,259 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:08,259 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:08,259 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:08,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 348ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1090514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:09,050 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629742_1099518554372] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 32268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 415116, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:09,050 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:09,050 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:09,050 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:09,050 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:09,050 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:09,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 346ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1159115, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:09,674 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 353ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1226702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:09,832 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629742_1099518554372] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 36562 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 497665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:09,832 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 3661 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 705677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:09,832 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:09,832 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:09,832 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:09,832 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:09,832 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:10,174 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 330ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005969, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:10,281 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:42:10,318 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:42:10,345 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024523027 with entries=78130, filesize=122.40 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024530281 2015-10-16 12:42:10,349 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629744_1099518554374] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1270861, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:10,674 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 379ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1209078, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:10,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629744_1099518554374] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 1806 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 634283, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:10,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 3885 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787202, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:10,730 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:10,730 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:10,730 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:10,730 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:10,730 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:11,175 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629743_1099518554373] hdfs.DFSClient: Slow ReadProcessor read fields took 357ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1231963, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:11,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629744_1099518554374] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 6166 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 529961, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:11,571 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:42:11,571 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:42:11,571 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:42:11,571 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:42:11,572 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:42:11,674 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629745_1099518554375] hdfs.DFSClient: Slow ReadProcessor read fields took 367ms (threshold=20ms); ack: seqno: 4181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1548158, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:12,174 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629745_1099518554375] hdfs.DFSClient: Slow ReadProcessor read fields took 382ms (threshold=20ms); ack: seqno: 4348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1261944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:12,359 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629744_1099518554374] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 10379 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 698884, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:12,359 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:12,359 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:12,359 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:12,359 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:12,359 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:12,674 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629745_1099518554375] hdfs.DFSClient: Slow ReadProcessor read fields took 368ms (threshold=20ms); ack: seqno: 4517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1116830, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:12,753 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#7 average throughput is 20.33 MB/sec, slept 27 time(s) and total slept time is 9837 ms. 0 active compactions remaining, total limit is 20.00 MB/sec 2015-10-16 12:42:12,767 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/eb66828d0bc2426d88885c2517929a18 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eb66828d0bc2426d88885c2517929a18 2015-10-16 12:42:12,775 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:42:12,776 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:42:12,785 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1bae3cb3341042cca6e270ad3b99890b, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1bae3cb3341042cca6e270ad3b99890b 2015-10-16 12:42:12,794 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5f90d9b1208e4e949b21247588f0a0e1, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5f90d9b1208e4e949b21247588f0a0e1 2015-10-16 12:42:12,802 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d8be11cac05a433db34a73b32597b3a4, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d8be11cac05a433db34a73b32597b3a4 2015-10-16 12:42:12,802 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into eb66828d0bc2426d88885c2517929a18(size=284.1 M), total size for store is 1.8 G. This selection was in queue for 0sec, and took 13sec to execute. 2015-10-16 12:42:12,802 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=284.1 M, priority=0, time=36902427992772353; duration=13sec 2015-10-16 12:42:12,802 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:42:13,059 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=326247776 > lower bound=16777216 2015-10-16 12:42:13,059 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=311.08 MB 2015-10-16 12:42:13,127 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629744_1099518554374] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 14190 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 806491, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:13,356 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629744_1099518554374] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 15344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 618157, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:13,356 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:13,356 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:13,356 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:13,356 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:13,356 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:14,107 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629746_1099518554376] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1039 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 727147, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:14,150 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629744_1099518554374] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 19037 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 730224, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:15,227 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1357871, memsize=311.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1bf8090d016c4878985c299b1671475c 2015-10-16 12:42:15,233 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1bf8090d016c4878985c299b1671475c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1bf8090d016c4878985c299b1671475c 2015-10-16 12:42:15,239 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1bf8090d016c4878985c299b1671475c, entries=1132850, sequenceid=1357871, filesize=172.1 M 2015-10-16 12:42:15,241 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~311.14 MB/326250240, currentsize=58.16 MB/60985680 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2182ms, sequenceid=1357871, compaction requested=true 2015-10-16 12:42:15,241 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 0 compacting, 10 eligible, 10 blocking 2015-10-16 12:42:15,242 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:42:15,246 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 552583729 starting at candidate #7 after considering 36 permutations with 36 in ratio 2015-10-16 12:42:15,246 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:42:15,246 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:42:15,246 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=527.0 M 2015-10-16 12:42:15,246 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eb66828d0bc2426d88885c2517929a18, keycount=1869900, bloomtype=ROW, size=284.1 M, encoding=NONE, seqNum=1197952 2015-10-16 12:42:15,246 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8beba6ad43e24a6fada0575aec4e0ded, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=1244581 2015-10-16 12:42:15,246 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1bf8090d016c4878985c299b1671475c, keycount=1132850, bloomtype=ROW, size=172.1 M, encoding=NONE, seqNum=1357871 2015-10-16 12:42:15,263 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:42:15,536 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629744_1099518554374] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 25998 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 686440, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:15,536 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:15,536 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:15,536 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:15,536 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:15,536 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:15,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629748_1099518554378] hdfs.DFSClient: Slow ReadProcessor read fields took 382ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1328005, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:42:16,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629748_1099518554378] hdfs.DFSClient: Slow ReadProcessor read fields took 388ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1436994, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:42:16,385 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629744_1099518554374] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 30348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 525741, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:16,385 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629748_1099518554378] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 726578, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:42:16,385 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:16,385 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:16,385 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:16,385 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:16,385 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:16,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629748_1099518554378] hdfs.DFSClient: Slow ReadProcessor read fields took 365ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1255868, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:42:17,279 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:42:17,279 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629744_1099518554374] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 35055 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 575431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:17,279 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:42:17,279 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:42:17,279 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:42:17,279 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:42:17,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629748_1099518554378] hdfs.DFSClient: Slow ReadProcessor read fields took 389ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1476522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:42:17,707 DEBUG [PriorityRpcServer.handler=10,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:42:17,707 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134286016 > lower bound=16777216 2015-10-16 12:42:17,707 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:42:17,732 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629744_1099518554374] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 37377 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 633079, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:17,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629748_1099518554378] hdfs.DFSClient: Slow ReadProcessor read fields took 384ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1298369, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:42:18,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629749_1099518554379] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 288 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 682832, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:18,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629744_1099518554374] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 38621 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 472600, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:18,002 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:18,002 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:18,002 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:18,002 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:18,034 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:42:18,086 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024530281 with entries=78006, filesize=122.21 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024538034 2015-10-16 12:42:18,088 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024515833 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024515833 2015-10-16 12:42:18,090 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629750_1099518554380] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026336, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:18,094 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024523027 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024523027 2015-10-16 12:42:18,281 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629748_1099518554378] hdfs.DFSClient: Slow ReadProcessor read fields took 379ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174826, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:42:18,498 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629750_1099518554380] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 1699 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 476058, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:18,498 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629749_1099518554379] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 961 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 818677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:18,498 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 69ms (threshold=20ms) 2015-10-16 12:42:18,498 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 69ms (threshold=20ms) 2015-10-16 12:42:18,498 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 69ms (threshold=20ms) 2015-10-16 12:42:18,498 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 69ms (threshold=20ms) 2015-10-16 12:42:18,627 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1404503, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/bd73bcdedaa245c98c1485faa64b2cd9 2015-10-16 12:42:18,635 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/bd73bcdedaa245c98c1485faa64b2cd9 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bd73bcdedaa245c98c1485faa64b2cd9 2015-10-16 12:42:18,648 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bd73bcdedaa245c98c1485faa64b2cd9, entries=466280, sequenceid=1404503, filesize=70.8 M 2015-10-16 12:42:18,650 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134285600, currentsize=22.92 MB/24035600 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 943ms, sequenceid=1404503, compaction requested=true 2015-10-16 12:42:18,650 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:42:18,650 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 3 compacting, 1 eligible, 10 blocking 2015-10-16 12:42:18,650 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:42:18,650 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:42:18,651 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:42:18,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629748_1099518554378] hdfs.DFSClient: Slow ReadProcessor read fields took 374ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1077934, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:42:19,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629748_1099518554378] hdfs.DFSClient: Slow ReadProcessor read fields took 384ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964475, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:42:19,506 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629750_1099518554380] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 6963 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 510915, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:19,506 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:19,506 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:19,506 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:19,506 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:19,506 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:19,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629748_1099518554378] hdfs.DFSClient: Slow ReadProcessor read fields took 382ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054718, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:42:20,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629748_1099518554378] hdfs.DFSClient: Slow ReadProcessor read fields took 385ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1035723, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:42:20,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629750_1099518554380] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 11525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 439045, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:20,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629748_1099518554378] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 1728 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 688186, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:42:20,361 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:20,361 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:20,361 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:20,361 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:20,661 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629750_1099518554380] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 12715 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 745629, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:20,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629748_1099518554378] hdfs.DFSClient: Slow ReadProcessor read fields took 346ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1064178, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:42:21,302 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629750_1099518554380] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 15948 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 430906, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:21,302 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:21,302 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:21,302 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:21,302 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:21,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629748_1099518554378] hdfs.DFSClient: Slow ReadProcessor read fields took 411ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081864, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:42:21,803 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629751_1099518554381] hdfs.DFSClient: Slow ReadProcessor read fields took 383ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1264040, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:22,069 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629750_1099518554380] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 20169 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 443518, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:22,069 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:22,069 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:22,069 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:22,069 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:22,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629751_1099518554381] hdfs.DFSClient: Slow ReadProcessor read fields took 378ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:22,328 DEBUG [PriorityRpcServer.handler=7,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:42:22,328 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:42:22,328 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:42:22,328 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 3 compacting, 1 eligible, 10 blocking 2015-10-16 12:42:22,329 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:42:22,329 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:42:22,329 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:42:22,803 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629751_1099518554381] hdfs.DFSClient: Slow ReadProcessor read fields took 381ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1025984, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:22,904 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629751_1099518554381] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 2611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 799356, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:22,904 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629750_1099518554380] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 24606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 643940, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:22,904 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:22,904 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:22,904 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:22,904 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:22,904 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:23,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629750_1099518554380] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 25474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1374866, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:23,135 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 81ms (threshold=20ms) 2015-10-16 12:42:23,135 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 83ms (threshold=20ms) 2015-10-16 12:42:23,185 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629750_1099518554380] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 25476 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 51750130, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:23,185 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 132ms (threshold=20ms) 2015-10-16 12:42:23,185 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 12:42:23,185 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 12:42:23,185 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 132ms (threshold=20ms) 2015-10-16 12:42:23,185 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 132ms (threshold=20ms) 2015-10-16 12:42:23,185 INFO [sync.3] wal.FSHLog: Slow sync cost: 132 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:23,185 INFO [sync.2] wal.FSHLog: Slow sync cost: 132 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:23,185 INFO [sync.4] wal.FSHLog: Slow sync cost: 132 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:23,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629751_1099518554381] hdfs.DFSClient: Slow ReadProcessor read fields took 354ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1104624, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:23,803 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629751_1099518554381] hdfs.DFSClient: Slow ReadProcessor read fields took 378ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010354, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:23,857 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629751_1099518554381] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 2874 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825823, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:23,857 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629750_1099518554380] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 29017 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 383793, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:23,857 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:23,857 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:23,857 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:23,857 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:24,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629751_1099518554381] hdfs.DFSClient: Slow ReadProcessor read fields took 350ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:24,660 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629750_1099518554380] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 33349 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 360862, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:24,660 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:24,660 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:24,660 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:24,660 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:24,660 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:24,803 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629751_1099518554381] hdfs.DFSClient: Slow ReadProcessor read fields took 381ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1041932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:25,075 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629750_1099518554380] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 35421 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 42666278, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:25,075 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:42:25,075 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:42:25,075 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:42:25,075 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:42:25,075 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:42:25,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629751_1099518554381] hdfs.DFSClient: Slow ReadProcessor read fields took 379ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1041406, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:25,488 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629750_1099518554380] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 37558 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 524312, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:25,488 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:25,488 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:25,488 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:25,488 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:25,488 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:25,727 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:42:25,766 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:25,777 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024538034 with entries=78053, filesize=122.28 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024545728 2015-10-16 12:42:25,803 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629751_1099518554381] hdfs.DFSClient: Slow ReadProcessor read fields took 382ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1165744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:26,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629751_1099518554381] hdfs.DFSClient: Slow ReadProcessor read fields took 380ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1313822, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:26,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629751_1099518554381] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 3718 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 743482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:26,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629752_1099518554382] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 3068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 450188, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:26,361 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:26,361 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:26,361 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:26,361 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:26,361 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:26,705 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629752_1099518554382] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 4914 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 600680, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:26,804 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629751_1099518554381] hdfs.DFSClient: Slow ReadProcessor read fields took 350ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1333796, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:26,821 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629752_1099518554382] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 5105 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 745747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:27,260 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629752_1099518554382] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 7428 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 597885, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:27,260 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:27,260 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:27,260 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:27,260 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:27,260 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:27,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629751_1099518554381] hdfs.DFSClient: Slow ReadProcessor read fields took 384ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1171678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:27,804 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629753_1099518554383] hdfs.DFSClient: Slow ReadProcessor read fields took 368ms (threshold=20ms); ack: seqno: 4182 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1346780, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:28,016 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629752_1099518554382] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 11586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 501042, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:28,016 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:28,016 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:28,016 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:28,017 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:28,017 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:28,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629753_1099518554383] hdfs.DFSClient: Slow ReadProcessor read fields took 387ms (threshold=20ms); ack: seqno: 4348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1460614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:28,804 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629753_1099518554383] hdfs.DFSClient: Slow ReadProcessor read fields took 378ms (threshold=20ms); ack: seqno: 4517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1398641, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:28,881 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:42:28,881 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:42:28,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629752_1099518554382] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 16222 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 567445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:28,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629753_1099518554383] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 4582 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 793275, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:28,881 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:42:28,881 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:42:29,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629753_1099518554383] hdfs.DFSClient: Slow ReadProcessor read fields took 349ms (threshold=20ms); ack: seqno: 4684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1385993, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:29,651 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629752_1099518554382] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 20677 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 484128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:29,651 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:29,651 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:29,651 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:29,651 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:29,651 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:42:29,805 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629753_1099518554383] hdfs.DFSClient: Slow ReadProcessor read fields took 381ms (threshold=20ms); ack: seqno: 4851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1337917, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:30,205 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629752_1099518554382] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 23538 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 613992, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:30,205 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:42:30,205 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:42:30,205 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:42:30,205 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:42:30,205 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:42:30,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629753_1099518554383] hdfs.DFSClient: Slow ReadProcessor read fields took 372ms (threshold=20ms); ack: seqno: 5018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1421531, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:30,476 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629752_1099518554382] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 24900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 552548, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:30,476 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:42:30,476 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:42:30,476 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:42:30,476 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:42:30,476 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:42:30,805 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629753_1099518554383] hdfs.DFSClient: Slow ReadProcessor read fields took 371ms (threshold=20ms); ack: seqno: 5185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1347754, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:31,306 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629753_1099518554383] hdfs.DFSClient: Slow ReadProcessor read fields took 377ms (threshold=20ms); ack: seqno: 5352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1434649, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:31,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629753_1099518554383] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 5386 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 697063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:31,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629752_1099518554382] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 29635 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 493969, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:31,358 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:31,358 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:31,358 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:31,358 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:31,806 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629753_1099518554383] hdfs.DFSClient: Slow ReadProcessor read fields took 348ms (threshold=20ms); ack: seqno: 5521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2026161, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:32,126 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629752_1099518554382] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 34016 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 431802, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:32,126 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:42:32,126 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:42:32,126 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:42:32,126 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:42:32,126 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:42:32,306 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629753_1099518554383] hdfs.DFSClient: Slow ReadProcessor read fields took 375ms (threshold=20ms); ack: seqno: 5688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1393378, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:32,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629752_1099518554382] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 35429 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:32,986 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629752_1099518554382] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 38564 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 445819, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:32,986 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:32,986 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:32,986 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:32,986 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:33,093 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:42:33,152 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024545728 with entries=78101, filesize=122.36 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024553093 2015-10-16 12:42:33,157 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1161402, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:33,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629753_1099518554383] hdfs.DFSClient: Slow ReadProcessor read fields took 888ms (threshold=20ms); ack: seqno: 5855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 498448503, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:33,847 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 3688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 433815, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:33,847 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:42:33,847 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:42:33,847 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:42:33,847 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:42:33,869 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629753_1099518554383] hdfs.DFSClient: Slow ReadProcessor read fields took 375ms (threshold=20ms); ack: seqno: 6189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111371, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:34,369 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629755_1099518554385] hdfs.DFSClient: Slow ReadProcessor read fields took 384ms (threshold=20ms); ack: seqno: 6358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1125362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:34,642 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:42:34,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 7933 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 577723, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:34,870 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629755_1099518554385] hdfs.DFSClient: Slow ReadProcessor read fields took 387ms (threshold=20ms); ack: seqno: 6525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1221305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:35,371 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629755_1099518554385] hdfs.DFSClient: Slow ReadProcessor read fields took 389ms (threshold=20ms); ack: seqno: 6694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1074992, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:35,490 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629755_1099518554385] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 6807 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 784569, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:35,490 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 12426 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 518810, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:35,491 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:42:35,491 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:42:35,491 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:42:35,491 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:42:35,491 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:42:35,871 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629755_1099518554385] hdfs.DFSClient: Slow ReadProcessor read fields took 338ms (threshold=20ms); ack: seqno: 6861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1465103, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:36,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629755_1099518554385] hdfs.DFSClient: Slow ReadProcessor read fields took 401ms (threshold=20ms); ack: seqno: 7028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1812031, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:36,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629755_1099518554385] hdfs.DFSClient: Slow ReadProcessor read fields took 403ms (threshold=20ms); ack: seqno: 7195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1640067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:37,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629755_1099518554385] hdfs.DFSClient: Slow ReadProcessor read fields took 401ms (threshold=20ms); ack: seqno: 7362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1654323, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:37,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629755_1099518554385] hdfs.DFSClient: Slow ReadProcessor read fields took 403ms (threshold=20ms); ack: seqno: 7529 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1981404, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:38,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629755_1099518554385] hdfs.DFSClient: Slow ReadProcessor read fields took 403ms (threshold=20ms); ack: seqno: 7696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1655189, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:38,460 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629755_1099518554385] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 7784 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 638286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 12:42:38,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629755_1099518554385] hdfs.DFSClient: Slow ReadProcessor read fields took 364ms (threshold=20ms); ack: seqno: 7863 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1682821, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:39,291 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#8 average throughput is 21.34 MB/sec, slept 46 time(s) and total slept time is 17323 ms. 0 active compactions remaining, total limit is unlimited 2015-10-16 12:42:39,309 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1bdc57b732a3479ab7614a76cbaba5bf as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1bdc57b732a3479ab7614a76cbaba5bf 2015-10-16 12:42:39,324 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 3608ms (threshold=20ms); ack: seqno: 13716 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026967, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:39,325 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:42:39,329 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:42:39,390 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eb66828d0bc2426d88885c2517929a18, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eb66828d0bc2426d88885c2517929a18 2015-10-16 12:42:39,401 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8beba6ad43e24a6fada0575aec4e0ded, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8beba6ad43e24a6fada0575aec4e0ded 2015-10-16 12:42:39,409 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1bf8090d016c4878985c299b1671475c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1bf8090d016c4878985c299b1671475c 2015-10-16 12:42:39,409 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 1bdc57b732a3479ab7614a76cbaba5bf(size=526.9 M), total size for store is 2.1 G. This selection was in queue for 0sec, and took 24sec to execute. 2015-10-16 12:42:39,410 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=527.0 M, priority=0, time=36902444099826937; duration=24sec 2015-10-16 12:42:39,410 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:42:39,431 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536920256 > lower bound=16777216 2015-10-16 12:42:39,431 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 12:42:39,541 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 216ms (threshold=20ms); ack: seqno: 13717 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:42,326 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1590944, memsize=512.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d2ffb009a2054a4ba60824614a3c2918 2015-10-16 12:42:42,334 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d2ffb009a2054a4ba60824614a3c2918 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d2ffb009a2054a4ba60824614a3c2918 2015-10-16 12:42:42,347 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d2ffb009a2054a4ba60824614a3c2918, entries=1864360, sequenceid=1590944, filesize=283.2 M 2015-10-16 12:42:42,349 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 2809ms (threshold=20ms); ack: seqno: 13718 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819125, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:42,349 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536919840, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2918ms, sequenceid=1590944, compaction requested=true 2015-10-16 12:42:42,349 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:42:42,350 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 0 compacting, 10 eligible, 10 blocking 2015-10-16 12:42:42,357 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 668827732 starting at candidate #0 after considering 36 permutations with 34 in ratio 2015-10-16 12:42:42,357 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:42:42,357 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:42:42,357 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=637.8 M 2015-10-16 12:42:42,357 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4463fa30fe63407d9f7adf901c26a347, keycount=1401170, bloomtype=ROW, size=212.2 M, encoding=NONE, seqNum=140130 2015-10-16 12:42:42,357 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2ab4a2928d1d4cdda3dacc21149f9a5c, keycount=1398910, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=280033 2015-10-16 12:42:42,357 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b4c8a0fc21be4114b78df88d24074046, keycount=1402660, bloomtype=ROW, size=213.1 M, encoding=NONE, seqNum=420311 2015-10-16 12:42:42,372 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:42:43,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629760_1099518554390] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 2082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 44560307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:42:44,041 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629761_1099518554391] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 2494 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801053, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:42:44,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 1714ms (threshold=20ms); ack: seqno: 13719 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:44,106 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 13721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 698753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:44,149 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 13735 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 543758, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:44,352 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 203ms (threshold=20ms); ack: seqno: 13736 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7544183, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:44,402 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 13739 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834670, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:44,450 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 13742 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 771378, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:44,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 154ms (threshold=20ms); ack: seqno: 13749 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 802371, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:44,809 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629761_1099518554391] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 3615 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 698551, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:42:44,809 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 14674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 579273, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:44,809 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:42:44,809 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:42:44,809 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:42:44,809 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:42:44,809 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:42:45,150 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629761_1099518554391] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 3980 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 716383, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:42:45,150 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 16207 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 530654, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:45,150 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:42:45,150 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 12:42:45,150 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:42:45,150 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:42:45,150 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:42:45,323 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629761_1099518554391] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 4166 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 19345667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:42:45,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629762_1099518554392] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 4287 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 884197, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:45,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 17444 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 592126, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:45,417 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:45,417 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:45,417 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:45,418 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:45,418 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:42:45,936 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 19971 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:45,936 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:45,936 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:45,936 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:45,936 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629762_1099518554392] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 5007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:45,936 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:46,446 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:46,446 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:46,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629762_1099518554392] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 5738 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 742962, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:46,446 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:46,446 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:46,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 22495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 502552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:46,447 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:46,900 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 24433 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 647679, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:46,961 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629762_1099518554392] hdfs.DFSClient: Slow ReadProcessor read fields took 170ms (threshold=20ms); ack: seqno: 6250 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 89725272, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:47,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 25158 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 559733, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:47,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629763_1099518554393] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 6426 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 674671, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:47,063 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:47,063 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:47,063 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:47,063 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:47,063 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:47,575 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 27696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 658537, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:47,575 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629763_1099518554393] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 7156 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729883, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:47,575 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:47,575 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:47,575 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:47,575 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:47,575 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:48,083 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 30268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 505459, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:48,083 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629763_1099518554393] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 7888 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 696321, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:48,083 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:48,083 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:48,083 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:48,083 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:48,084 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:48,597 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629764_1099518554394] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 8586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 662762, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:42:48,597 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 32966 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 531574, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:48,597 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:48,597 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:48,597 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:48,597 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:48,597 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:49,110 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 35553 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 677211, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:49,110 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629764_1099518554394] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 9300 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 739829, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:42:49,110 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:49,110 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:49,110 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:49,110 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:49,110 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:49,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 570ms (threshold=20ms); ack: seqno: 36781 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 569689087, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:49,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629764_1099518554394] hdfs.DFSClient: Slow ReadProcessor read fields took 570ms (threshold=20ms); ack: seqno: 9654 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 570356667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:42:49,916 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 570ms (threshold=20ms) 2015-10-16 12:42:49,916 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 570ms (threshold=20ms) 2015-10-16 12:42:49,916 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 570ms (threshold=20ms) 2015-10-16 12:42:49,916 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 571ms (threshold=20ms) 2015-10-16 12:42:49,916 INFO [sync.3] wal.FSHLog: Slow sync cost: 570 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:49,916 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 571ms (threshold=20ms) 2015-10-16 12:42:49,916 INFO [sync.2] wal.FSHLog: Slow sync cost: 570 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:49,916 INFO [sync.1] wal.FSHLog: Slow sync cost: 570 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:49,916 INFO [sync.4] wal.FSHLog: Slow sync cost: 570 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:49,916 INFO [sync.0] wal.FSHLog: Slow sync cost: 570 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:49,957 DEBUG [PriorityRpcServer.handler=4,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:42:49,957 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134261696 > lower bound=16777216 2015-10-16 12:42:49,958 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:42:49,984 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 36987 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 609758, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:50,156 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:50,156 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629764_1099518554394] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 10068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 768247, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:42:50,156 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629754_1099518554384] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 37747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 381299, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:50,157 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:50,157 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:50,157 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:50,157 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:50,376 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#9 average throughput is 77.60 MB/sec, slept 0 time(s) and total slept time is 0 ms. 0 active compactions remaining, total limit is unlimited 2015-10-16 12:42:50,392 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:42:50,403 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629764_1099518554394] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 10381 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 22031881, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:42:50,403 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:50,432 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:42:50,440 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/4b24e48b93884d2ca3d87733ccadebc7 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4b24e48b93884d2ca3d87733ccadebc7 2015-10-16 12:42:50,459 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024553093 with entries=78099, filesize=122.35 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024570392 2015-10-16 12:42:50,461 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024530281 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024530281 2015-10-16 12:42:50,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1129341, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:50,463 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:42:50,464 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:42:50,467 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024538034 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024538034 2015-10-16 12:42:50,475 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4463fa30fe63407d9f7adf901c26a347, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4463fa30fe63407d9f7adf901c26a347 2015-10-16 12:42:50,484 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024545728 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024545728 2015-10-16 12:42:50,491 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2ab4a2928d1d4cdda3dacc21149f9a5c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2ab4a2928d1d4cdda3dacc21149f9a5c 2015-10-16 12:42:50,508 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b4c8a0fc21be4114b78df88d24074046, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b4c8a0fc21be4114b78df88d24074046 2015-10-16 12:42:50,508 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 4b24e48b93884d2ca3d87733ccadebc7(size=637.8 M), total size for store is 2.4 G. This selection was in queue for 0sec, and took 8sec to execute. 2015-10-16 12:42:50,508 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=637.8 M, priority=0, time=36902471210951240; duration=8sec 2015-10-16 12:42:50,508 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:42:50,992 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1637574, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8e531ef93b9149a3ba1f388efcc83d5c 2015-10-16 12:42:50,997 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8e531ef93b9149a3ba1f388efcc83d5c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8e531ef93b9149a3ba1f388efcc83d5c 2015-10-16 12:42:51,003 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8e531ef93b9149a3ba1f388efcc83d5c, entries=466260, sequenceid=1637574, filesize=70.8 M 2015-10-16 12:42:51,011 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134278560, currentsize=28.13 MB/29492720 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1054ms, sequenceid=1637574, compaction requested=true 2015-10-16 12:42:51,011 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:42:51,011 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 0 compacting, 9 eligible, 10 blocking 2015-10-16 12:42:51,015 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 9 files of size 2607642824 starting at candidate #0 after considering 28 permutations with 23 in ratio 2015-10-16 12:42:51,015 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction (all files) 2015-10-16 12:42:51,015 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:42:51,015 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 9 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=2.4 G 2015-10-16 12:42:51,016 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4b24e48b93884d2ca3d87733ccadebc7, keycount=4202740, bloomtype=ROW, size=637.8 M, encoding=NONE, seqNum=420311, earliestPutTs=1445024396631 2015-10-16 12:42:51,016 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0b7364a0f78146b9baca89016d5c720b, keycount=1398950, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=560219, earliestPutTs=1445024443101 2015-10-16 12:42:51,017 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0261c61815d0485497e8ee387c903d48, keycount=1398890, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=700121, earliestPutTs=1445024456072 2015-10-16 12:42:51,018 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/47014a9030d5469e9c456de21f92c5c4, keycount=1691380, bloomtype=ROW, size=257.0 M, encoding=NONE, seqNum=869273, earliestPutTs=1445024469352 2015-10-16 12:42:51,018 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5a28df304f4c441b9194a01700d939cf, keycount=1416620, bloomtype=ROW, size=215.2 M, encoding=NONE, seqNum=1010948, earliestPutTs=1445024485260 2015-10-16 12:42:51,018 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1bdc57b732a3479ab7614a76cbaba5bf, keycount=3469000, bloomtype=ROW, size=526.9 M, encoding=NONE, seqNum=1357871, earliestPutTs=1445024499555 2015-10-16 12:42:51,019 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bd73bcdedaa245c98c1485faa64b2cd9, keycount=466280, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=1404503, earliestPutTs=1445024533059 2015-10-16 12:42:51,019 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d2ffb009a2054a4ba60824614a3c2918, keycount=1864360, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=1590944, earliestPutTs=1445024537707 2015-10-16 12:42:51,020 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8e531ef93b9149a3ba1f388efcc83d5c, keycount=466260, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=1637574, earliestPutTs=1445024564061 2015-10-16 12:42:51,061 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:42:51,376 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:51,376 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 4762 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 565522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:51,376 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:51,376 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:51,933 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:42:51,933 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:42:51,933 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629767_1099518554397] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 971 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769147, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:51,935 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 7714 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1522876, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:52,475 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:52,475 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 10606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 374714, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:52,475 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629767_1099518554397] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1609 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 752607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:42:52,476 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:52,475 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:52,475 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:52,929 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 12566 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 622670, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:53,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629768_1099518554398] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 2344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 788132, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:42:53,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 13190 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 548453, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:53,066 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:53,066 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:53,066 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:53,066 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:53,066 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:53,575 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 15919 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 556000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:53,575 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629768_1099518554398] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 3028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 750627, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:42:53,575 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:53,575 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:53,575 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:53,575 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:53,575 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:54,080 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 18595 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 508807, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:54,080 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629768_1099518554398] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 3727 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 704376, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:42:54,080 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:54,080 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:54,080 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:54,080 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:54,080 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:54,572 DEBUG [PriorityRpcServer.handler=4,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:42:54,572 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134284096 > lower bound=16777216 2015-10-16 12:42:54,573 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:42:54,606 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629769_1099518554399] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 4465 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 652199, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:42:54,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 21136 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 530872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:55,041 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:42:55,041 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 23134 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 477637, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:55,042 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:55,042 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:55,042 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:55,042 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:42:55,043 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629770_1099518554400] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 556 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930963, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:42:55,047 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629769_1099518554399] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 5054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5502544, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:42:55,486 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629770_1099518554400] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 1083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839745, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:42:55,486 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 25237 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 600086, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:55,486 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629769_1099518554399] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 5649 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 807133, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:42:55,486 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:55,486 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:55,486 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:55,486 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:55,486 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:42:55,550 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1684214, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/4beb8db3487d4195a34893d32ddbaed3 2015-10-16 12:42:55,556 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/4beb8db3487d4195a34893d32ddbaed3 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4beb8db3487d4195a34893d32ddbaed3 2015-10-16 12:42:55,570 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4beb8db3487d4195a34893d32ddbaed3, entries=466350, sequenceid=1684214, filesize=70.8 M 2015-10-16 12:42:55,572 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134303840, currentsize=25.21 MB/26432160 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1000ms, sequenceid=1684214, compaction requested=false 2015-10-16 12:42:55,801 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 26566 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 12140954, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:55,801 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629769_1099518554399] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 5999 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 11896287, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:42:55,802 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 65ms (threshold=20ms) 2015-10-16 12:42:55,802 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 65ms (threshold=20ms) 2015-10-16 12:42:55,802 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 66ms (threshold=20ms) 2015-10-16 12:42:55,802 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 66ms (threshold=20ms) 2015-10-16 12:42:55,802 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 65ms (threshold=20ms) 2015-10-16 12:42:56,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629771_1099518554401] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 6362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3005689, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:56,063 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:56,064 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:56,064 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:56,064 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:56,064 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:56,577 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:56,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629771_1099518554401] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 7089 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:56,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 30300 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 586637, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:56,577 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:56,577 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:56,578 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:56,578 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:57,086 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 32834 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 621925, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:57,086 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629771_1099518554401] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 7817 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787784, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:42:57,086 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:57,086 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:57,086 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:57,087 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:57,610 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 35469 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 498969, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:57,610 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629772_1099518554402] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 8534 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 716461, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:42:57,610 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:57,611 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:58,149 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 38027 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 682681, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:58,149 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629772_1099518554402] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 9255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 744813, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:42:58,149 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:58,149 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:58,149 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:58,149 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:58,149 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:58,356 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629766_1099518554396] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 38667 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 708016, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:42:58,364 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:42:58,417 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024570392 with entries=77978, filesize=122.16 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024578364 2015-10-16 12:42:58,418 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024553093 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024553093 2015-10-16 12:42:58,420 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1039541, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:58,719 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1571 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2501891, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:58,719 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629772_1099518554402] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 10011 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2326905, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:42:58,719 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:58,719 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:58,719 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:58,719 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:58,719 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:42:59,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 4342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 557937, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:59,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629774_1099518554404] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 10683 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 875266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:42:59,246 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:59,246 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:59,246 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:59,246 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:59,246 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:42:59,441 DEBUG [PriorityRpcServer.handler=9,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:42:59,441 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134266896 > lower bound=16777216 2015-10-16 12:42:59,449 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.18 MB 2015-10-16 12:42:59,473 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 5461 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 523943, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:42:59,726 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:59,726 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:42:59,726 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:43:00,345 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629775_1099518554405] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 1153 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 63350889, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:00,345 WARN [MemStoreFlusher.1] hdfs.DFSClient: Slow waitForAckedSeqno took 65ms (threshold=20ms) 2015-10-16 12:43:00,349 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1730906, memsize=128.2 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/060a30edd42641ceb5cf26ada00bd31e 2015-10-16 12:43:00,357 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/060a30edd42641ceb5cf26ada00bd31e as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/060a30edd42641ceb5cf26ada00bd31e 2015-10-16 12:43:00,370 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/060a30edd42641ceb5cf26ada00bd31e, entries=466880, sequenceid=1730906, filesize=70.9 M 2015-10-16 12:43:00,373 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.23 MB/134456560, currentsize=23.34 MB/24476560 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 923ms, sequenceid=1730906, compaction requested=false 2015-10-16 12:43:00,678 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 11166 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 544297, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:43:00,678 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:43:00,678 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:43:00,678 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:43:00,678 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:43:00,678 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:43:01,210 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629776_1099518554406] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 13224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905070, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:43:01,210 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 13797 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 661864, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:43:01,210 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:01,210 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:01,210 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:01,210 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:01,211 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:01,746 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:01,746 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:01,746 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 16368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 495580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:43:01,746 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629776_1099518554406] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 13944 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 760313, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:43:01,747 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:01,747 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:01,747 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:02,276 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629777_1099518554407] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 14649 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1147540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:43:02,276 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 19055 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 463228, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:43:02,276 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:02,276 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:43:02,276 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:43:02,276 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:02,276 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:43:02,816 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 21880 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 601290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:43:02,816 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629777_1099518554407] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 15311 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833163, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:43:02,816 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:02,816 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:02,816 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:02,816 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:02,816 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:03,352 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 24775 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 548595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:43:03,352 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629777_1099518554407] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 15963 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 711626, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:43:03,352 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:03,352 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:03,352 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:03,352 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:03,352 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:03,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 24809 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 899006, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:43:03,777 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629777_1099518554407] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 16445 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 850107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:43:03,777 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 26311 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 655488, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:43:03,778 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 67ms (threshold=20ms) 2015-10-16 12:43:03,778 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 67ms (threshold=20ms) 2015-10-16 12:43:03,778 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 67ms (threshold=20ms) 2015-10-16 12:43:03,778 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 67ms (threshold=20ms) 2015-10-16 12:43:03,778 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 67ms (threshold=20ms) 2015-10-16 12:43:03,998 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629778_1099518554408] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 16678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3290540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:43:03,998 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 27391 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 597028, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:43:03,998 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:03,998 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:43:03,998 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:03,999 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:03,998 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:04,267 DEBUG [PriorityRpcServer.handler=12,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:43:04,267 WARN [MemStoreFlusher.0] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:43:04,267 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:43:04,268 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 9 compacting, 2 eligible, 10 blocking 2015-10-16 12:43:04,269 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:43:04,269 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:43:04,269 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:43:04,558 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629778_1099518554408] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 17335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787198, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:43:04,558 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 30326 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 422109, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:43:04,558 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:43:04,558 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:43:04,558 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:43:04,558 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:43:04,559 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:43:05,091 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629778_1099518554408] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 18067 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 748865, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:43:05,091 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 32843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 897223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:43:05,092 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:43:05,092 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:05,092 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:43:05,092 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:43:05,092 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:05,630 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 35511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 460561, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:43:05,630 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:05,630 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:05,630 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:05,630 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:05,630 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:06,145 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 38283 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 668144, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:43:06,145 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629779_1099518554409] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 19447 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864400, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:43:06,145 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:06,145 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:06,145 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:06,145 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:06,145 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:06,277 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:43:06,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629779_1099518554409] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 20123 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 856870, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:43:06,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629773_1099518554403] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 41068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 484360, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:43:06,690 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:06,690 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:06,690 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:06,690 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:06,690 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:07,285 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629779_1099518554409] hdfs.DFSClient: Slow ReadProcessor read fields took 141ms (threshold=20ms); ack: seqno: 20838 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 140709468, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:43:07,289 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 1006ms (threshold=20ms) 2015-10-16 12:43:07,294 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 599ms (threshold=20ms) 2015-10-16 12:43:07,294 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 600ms (threshold=20ms) 2015-10-16 12:43:07,294 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 601ms (threshold=20ms) 2015-10-16 12:43:07,294 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 601ms (threshold=20ms) 2015-10-16 12:43:07,294 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 600ms (threshold=20ms) 2015-10-16 12:43:07,294 INFO [sync.1] wal.FSHLog: Slow sync cost: 600 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:43:07,294 INFO [sync.3] wal.FSHLog: Slow sync cost: 600 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:43:07,294 INFO [sync.4] wal.FSHLog: Slow sync cost: 600 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:43:07,294 INFO [sync.0] wal.FSHLog: Slow sync cost: 600 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:43:07,299 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Slow sync cost: 1016 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:43:07,307 INFO [sync.2] wal.FSHLog: Slow sync cost: 613 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:43:07,316 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024578364 with entries=81724, filesize=128.03 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024586277 2015-10-16 12:43:07,319 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024570392 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024570392 2015-10-16 12:43:07,320 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1173598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:07,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629782_1099518554412] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 21197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 806684, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:43:07,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 1130 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 427150, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:07,534 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:07,534 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:07,534 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:07,534 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:07,534 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:08,038 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:08,039 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 3965 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 552131, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:08,039 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:08,039 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629782_1099518554412] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 21852 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 763988, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:43:08,557 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629782_1099518554412] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 22511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 714339, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:43:08,557 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 6759 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 579706, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:08,557 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:08,557 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:08,557 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:08,557 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:08,557 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:08,848 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 7914 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:09,136 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629783_1099518554413] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 23252 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913022, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:43:09,137 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 9308 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 572777, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:09,137 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:09,137 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:09,137 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:09,137 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:09,137 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:09,502 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629783_1099518554413] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 23633 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797116, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:43:09,502 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 10946 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 474201, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:09,502 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 12:43:09,502 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 12:43:09,503 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:43:09,503 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 12:43:09,503 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 12:43:09,756 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629783_1099518554413] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 23900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 871119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:43:09,756 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 12152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 688013, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:09,757 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:09,757 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:09,757 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:09,757 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:09,757 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:10,292 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629783_1099518554413] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 24570 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758121, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:43:10,292 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 14994 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 426643, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:10,292 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:10,293 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:10,293 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:10,819 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:10,819 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 17697 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 449918, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:10,819 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:10,819 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:10,819 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:10,820 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629784_1099518554414] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 25269 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913845, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:11,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629784_1099518554414] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 25932 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 759726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:11,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 20579 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 502574, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:11,358 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:11,358 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:11,358 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:11,358 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:11,358 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:11,894 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629784_1099518554414] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 26589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 879813, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:11,894 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 23469 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 528689, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:11,894 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:11,894 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:11,894 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:11,895 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:11,895 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:12,421 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 26406 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 465220, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:12,421 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629785_1099518554415] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 27240 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 735361, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:43:12,421 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:12,421 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:12,421 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:12,421 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:12,421 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:12,961 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629785_1099518554415] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 27897 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701817, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:43:12,961 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 29331 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 525826, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:12,961 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:12,961 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:12,961 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:12,961 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:12,961 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:13,378 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 31176 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 677030, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:13,539 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 31919 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 442541, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:13,539 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629785_1099518554415] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 28614 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 745227, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:43:13,539 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:13,539 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:13,539 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:13,539 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:13,539 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:14,073 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 34787 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 550889, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:14,073 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629786_1099518554416] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 29284 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 809353, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:43:14,074 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:14,074 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:14,074 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:14,074 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:14,074 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:14,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629780_1099518554410] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 37587 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 546330, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:43:14,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629786_1099518554416] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 29961 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758738, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:43:14,608 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:14,608 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:14,608 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:14,608 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:14,609 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:14,940 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:43:14,974 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:14,999 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024586277 with entries=78098, filesize=122.35 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024594940 2015-10-16 12:43:15,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932425, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:15,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 590 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 566654, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:15,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629786_1099518554416] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 30659 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811751, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:43:15,135 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:15,135 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:15,135 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:15,135 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:15,135 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:15,668 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 3432 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 406299, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:15,668 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:15,668 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:15,668 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:15,668 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:15,668 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:43:16,409 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:16,410 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629788_1099518554418] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 31693 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 703199, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:43:16,410 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 7432 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 477747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:16,410 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:16,410 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:16,930 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629788_1099518554418] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 32373 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 734839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:43:16,930 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 10205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 483031, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:16,930 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:16,930 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:16,930 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:16,930 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:16,930 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:17,439 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:17,439 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629788_1099518554418] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 33045 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 749457, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:43:17,439 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 13023 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 621406, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:17,439 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:17,439 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:17,439 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:17,439 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:17,653 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 13746 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 685596, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:18,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 15587 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 396550, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:18,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629789_1099518554419] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 33776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 711093, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:18,019 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:18,019 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:18,019 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:43:18,019 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:18,019 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:43:18,585 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:18,585 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:18,585 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:18,586 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 18326 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 564315, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:18,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629789_1099518554419] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 34468 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915572, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:19,088 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629789_1099518554419] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 35104 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 713847, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:19,438 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629790_1099518554420] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 35619 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672742, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:43:20,299 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629790_1099518554420] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 36975 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 718224, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:43:21,156 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629791_1099518554421] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 38330 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 745225, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:43:22,088 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629792_1099518554422] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 39695 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 705815, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:43:22,572 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#10 average throughput is 76.77 MB/sec, slept 0 time(s) and total slept time is 0 ms. 0 active compactions remaining, total limit is unlimited 2015-10-16 12:43:22,600 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/53d2a7c0edf3492cadeec7d370728efb as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/53d2a7c0edf3492cadeec7d370728efb 2015-10-16 12:43:22,615 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 3905ms (threshold=20ms); ack: seqno: 19012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905590, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:22,615 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:43:22,618 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:43:22,623 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4b24e48b93884d2ca3d87733ccadebc7, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4b24e48b93884d2ca3d87733ccadebc7 2015-10-16 12:43:22,631 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0b7364a0f78146b9baca89016d5c720b, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0b7364a0f78146b9baca89016d5c720b 2015-10-16 12:43:22,640 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0261c61815d0485497e8ee387c903d48, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0261c61815d0485497e8ee387c903d48 2015-10-16 12:43:22,648 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/47014a9030d5469e9c456de21f92c5c4, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/47014a9030d5469e9c456de21f92c5c4 2015-10-16 12:43:22,656 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5a28df304f4c441b9194a01700d939cf, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5a28df304f4c441b9194a01700d939cf 2015-10-16 12:43:22,681 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1bdc57b732a3479ab7614a76cbaba5bf, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1bdc57b732a3479ab7614a76cbaba5bf 2015-10-16 12:43:22,690 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bd73bcdedaa245c98c1485faa64b2cd9, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bd73bcdedaa245c98c1485faa64b2cd9 2015-10-16 12:43:22,698 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d2ffb009a2054a4ba60824614a3c2918, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d2ffb009a2054a4ba60824614a3c2918 2015-10-16 12:43:22,706 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8e531ef93b9149a3ba1f388efcc83d5c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8e531ef93b9149a3ba1f388efcc83d5c 2015-10-16 12:43:22,706 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 9 (all) file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 53d2a7c0edf3492cadeec7d370728efb(size=2.4 G), total size for store is 2.6 G. This selection was in queue for 0sec, and took 31sec to execute. 2015-10-16 12:43:22,706 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=9, fileSize=2.4 G, priority=2, time=36902479869193563; duration=31sec 2015-10-16 12:43:22,707 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:43:23,171 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536925216 > lower bound=16777216 2015-10-16 12:43:23,171 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 12:43:23,279 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 664ms (threshold=20ms); ack: seqno: 19013 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 933353, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:26,156 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1917349, memsize=512.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/bbfda5027a904d2993e04ab79313663f 2015-10-16 12:43:26,164 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/bbfda5027a904d2993e04ab79313663f as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bbfda5027a904d2993e04ab79313663f 2015-10-16 12:43:26,178 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bbfda5027a904d2993e04ab79313663f, entries=1864380, sequenceid=1917349, filesize=283.2 M 2015-10-16 12:43:26,179 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 2900ms (threshold=20ms); ack: seqno: 19014 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 574246, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:26,180 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536924800, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 3009ms, sequenceid=1917349, compaction requested=true 2015-10-16 12:43:26,180 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:43:26,180 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 4 store files, 0 compacting, 4 eligible, 10 blocking 2015-10-16 12:43:26,182 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio 2015-10-16 12:43:26,182 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:43:26,182 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:43:26,825 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 646ms (threshold=20ms); ack: seqno: 19015 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830712, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:26,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 19017 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 45756769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:26,875 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:26,876 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:26,876 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:43:26,876 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:43:26,876 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:43:27,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 23503 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 517545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:27,641 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:27,641 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:27,641 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:27,641 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:27,641 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:28,599 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 29098 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 503085, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:28,599 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:28,599 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:28,599 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:28,599 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:28,599 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:29,638 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 34502 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 675344, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:29,687 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629787_1099518554417] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 34601 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 531704, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:29,687 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:43:29,687 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:43:29,687 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:43:29,687 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:43:29,688 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:43:30,445 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:43:30,489 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024594940 with entries=78026, filesize=122.24 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024610446 2015-10-16 12:43:30,491 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024578364 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024578364 2015-10-16 12:43:30,492 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629796_1099518554426] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1015554, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:30,498 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024586277 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024586277 2015-10-16 12:43:30,734 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629796_1099518554426] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 1036 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 547101, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:30,734 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:43:30,734 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:43:30,734 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:43:30,734 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:43:31,087 DEBUG [PriorityRpcServer.handler=8,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:43:31,087 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134274256 > lower bound=16777216 2015-10-16 12:43:31,087 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:43:31,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629796_1099518554426] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 3093 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 508675, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:31,556 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629796_1099518554426] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 5302 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 545328, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:31,556 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629797_1099518554427] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 633 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 646338, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:43:31,556 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:31,557 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:31,556 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:31,556 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:31,556 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:31,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1963978, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/27bc39bdbe8c495eb24a465bb726f632 2015-10-16 12:43:31,888 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/27bc39bdbe8c495eb24a465bb726f632 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/27bc39bdbe8c495eb24a465bb726f632 2015-10-16 12:43:31,901 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/27bc39bdbe8c495eb24a465bb726f632, entries=466250, sequenceid=1963978, filesize=70.8 M 2015-10-16 12:43:31,904 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134276720, currentsize=22.70 MB/23805760 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 816ms, sequenceid=1963978, compaction requested=true 2015-10-16 12:43:31,904 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:43:31,904 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 5 store files, 0 compacting, 5 eligible, 10 blocking 2015-10-16 12:43:31,906 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 6 permutations with 0 in ratio 2015-10-16 12:43:31,906 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:43:31,906 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:43:32,416 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629796_1099518554426] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 9800 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 385580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:32,417 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:32,417 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:32,417 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:32,417 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:32,418 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:33,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629796_1099518554426] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 15376 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 653730, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:33,431 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:33,431 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:33,431 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:33,431 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:33,431 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:33,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629796_1099518554426] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 15415 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2173063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:34,594 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629796_1099518554426] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 20999 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 607437, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:34,594 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:43:34,594 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:43:34,594 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:43:34,595 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:43:34,596 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:43:35,628 DEBUG [PriorityRpcServer.handler=15,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:43:35,628 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134284864 > lower bound=16777216 2015-10-16 12:43:35,628 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:43:35,656 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629796_1099518554426] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 26496 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 547864, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:35,701 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629796_1099518554426] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 26566 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 561511, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:35,701 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:35,701 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:35,701 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:35,701 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:35,701 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:36,398 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629798_1099518554428] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 1061 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 682930, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:43:36,400 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629796_1099518554426] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 30120 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804853, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:36,472 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2010614, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/fdb2b87d945a423f8f16dd54d6226e1f 2015-10-16 12:43:36,478 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/fdb2b87d945a423f8f16dd54d6226e1f as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fdb2b87d945a423f8f16dd54d6226e1f 2015-10-16 12:43:36,484 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fdb2b87d945a423f8f16dd54d6226e1f, entries=466320, sequenceid=2010614, filesize=70.8 M 2015-10-16 12:43:36,485 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134297120, currentsize=22.56 MB/23660880 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 857ms, sequenceid=2010614, compaction requested=true 2015-10-16 12:43:36,486 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:43:36,486 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 6 store files, 0 compacting, 6 eligible, 10 blocking 2015-10-16 12:43:36,488 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 5 files of size 594207515 starting at candidate #1 after considering 10 permutations with 1 in ratio 2015-10-16 12:43:36,488 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:43:36,489 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:43:36,489 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=566.7 M 2015-10-16 12:43:36,489 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4beb8db3487d4195a34893d32ddbaed3, keycount=466350, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=1684214 2015-10-16 12:43:36,489 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/060a30edd42641ceb5cf26ada00bd31e, keycount=466880, bloomtype=ROW, size=70.9 M, encoding=NONE, seqNum=1730906 2015-10-16 12:43:36,489 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bbfda5027a904d2993e04ab79313663f, keycount=1864380, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=1917349 2015-10-16 12:43:36,489 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/27bc39bdbe8c495eb24a465bb726f632, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=1963978 2015-10-16 12:43:36,489 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fdb2b87d945a423f8f16dd54d6226e1f, keycount=466320, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=2010614 2015-10-16 12:43:36,517 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:43:36,993 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629796_1099518554426] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 32932 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 598383, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:36,993 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629799_1099518554429] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 598 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:36,993 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:36,993 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:36,993 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:36,993 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:37,410 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629796_1099518554426] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 34657 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922608, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:37,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629796_1099518554426] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 35382 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 463748, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:37,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629799_1099518554429] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 1351 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 768902, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:37,577 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:37,577 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:37,577 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:37,577 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:37,577 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:38,122 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629796_1099518554426] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 38077 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 399678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:38,122 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629799_1099518554429] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 2049 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833637, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:38,123 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:38,123 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:38,123 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:38,123 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:38,123 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:38,280 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:43:38,322 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024610446 with entries=77967, filesize=122.15 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024618280 2015-10-16 12:43:38,324 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024594940 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024594940 2015-10-16 12:43:38,326 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629801_1099518554431] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014630, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:38,664 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629801_1099518554431] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 1693 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 545923, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:38,664 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629800_1099518554430] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 2736 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 703783, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:38,664 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:38,664 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:38,664 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:38,664 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:38,664 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.42857142857142855, tune compaction throughput to 14.29 MB/sec 2015-10-16 12:43:39,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629800_1099518554430] hdfs.DFSClient: Slow ReadProcessor read fields took 574ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1344113, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:39,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629801_1099518554431] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 5935 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 567402, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:39,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629800_1099518554430] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 3070 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 809589, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:39,512 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:39,512 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:39,512 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:39,512 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:39,512 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:40,142 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629800_1099518554430] hdfs.DFSClient: Slow ReadProcessor read fields took 551ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1042564, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:40,351 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629801_1099518554431] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 10453 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 685040, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:40,351 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:40,351 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:40,351 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:40,351 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:40,351 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:43:40,409 DEBUG [PriorityRpcServer.handler=8,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:43:40,409 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134280176 > lower bound=16777216 2015-10-16 12:43:40,409 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:43:40,433 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629801_1099518554431] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 10787 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 597932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:40,844 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629800_1099518554430] hdfs.DFSClient: Slow ReadProcessor read fields took 577ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066939, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:41,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629800_1099518554430] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 3493 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 764948, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:41,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629801_1099518554431] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 13723 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 554436, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:41,027 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:41,027 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:41,027 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:41,027 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:41,028 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:41,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629802_1099518554432] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 830 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 823804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:43:41,187 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629801_1099518554431] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 14077 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 730849, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:41,247 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2057246, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/84cf48c60aeb42c594466cf9a26149f5 2015-10-16 12:43:41,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/84cf48c60aeb42c594466cf9a26149f5 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/84cf48c60aeb42c594466cf9a26149f5 2015-10-16 12:43:41,267 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/84cf48c60aeb42c594466cf9a26149f5, entries=466280, sequenceid=2057246, filesize=70.8 M 2015-10-16 12:43:41,269 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134282640, currentsize=20.78 MB/21789360 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 860ms, sequenceid=2057246, compaction requested=false 2015-10-16 12:43:41,543 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629800_1099518554430] hdfs.DFSClient: Slow ReadProcessor read fields took 498ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1034813, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:41,924 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629801_1099518554431] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 17932 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 396759, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:41,924 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:41,925 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:41,925 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:41,925 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:42,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629800_1099518554430] hdfs.DFSClient: Slow ReadProcessor read fields took 580ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1215619, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:42,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629801_1099518554431] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 22812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 504319, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:42,881 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:43:42,881 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:42,881 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:42,881 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:42,881 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:43:42,943 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629800_1099518554430] hdfs.DFSClient: Slow ReadProcessor read fields took 580ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089917, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:43,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629800_1099518554430] hdfs.DFSClient: Slow ReadProcessor read fields took 585ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1116421, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:43,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629801_1099518554431] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 27395 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 560565, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:43,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629800_1099518554430] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 4095 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 761966, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:43,731 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:43,731 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:43,731 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:43,731 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:43,731 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:44,343 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 543ms (threshold=20ms); ack: seqno: 4181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1664859, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:44,640 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629801_1099518554431] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 31877 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 581563, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:44,678 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629801_1099518554431] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 31901 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 566518, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:44,678 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:43:44,678 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:43:44,678 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:43:44,678 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:43:44,678 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:43:45,043 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 584ms (threshold=20ms); ack: seqno: 4348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1312385, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:45,103 DEBUG [PriorityRpcServer.handler=5,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:43:45,104 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134274256 > lower bound=16777216 2015-10-16 12:43:45,104 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:43:45,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629801_1099518554431] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 34169 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 539339, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:45,480 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629801_1099518554431] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 35857 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 463243, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:43:45,480 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629804_1099518554434] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 469 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769525, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:45,480 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:45,480 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:45,480 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:45,480 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:45,480 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:45,743 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 578ms (threshold=20ms); ack: seqno: 4517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1074755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:45,922 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2103882, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ea89219e65e84607a0ad2cb5120b6c9a 2015-10-16 12:43:45,927 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ea89219e65e84607a0ad2cb5120b6c9a as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ea89219e65e84607a0ad2cb5120b6c9a 2015-10-16 12:43:45,933 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ea89219e65e84607a0ad2cb5120b6c9a, entries=466320, sequenceid=2103882, filesize=70.8 M 2015-10-16 12:43:45,938 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134296880, currentsize=22.20 MB/23280720 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 834ms, sequenceid=2103882, compaction requested=true 2015-10-16 12:43:45,938 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:43:45,938 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 5 compacting, 2 eligible, 10 blocking 2015-10-16 12:43:45,939 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:43:45,939 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:43:45,939 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:43:46,071 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:43:46,122 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024618280 with entries=77968, filesize=122.16 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024626072 2015-10-16 12:43:46,124 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024610446 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024610446 2015-10-16 12:43:46,125 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629805_1099518554435] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1046243, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:46,203 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629805_1099518554435] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 321 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 717857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:46,444 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 570ms (threshold=20ms); ack: seqno: 4684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1387465, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:47,056 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629805_1099518554435] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 5227 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 453249, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:47,056 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:47,056 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:47,056 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:47,056 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:47,056 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:47,144 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 573ms (threshold=20ms); ack: seqno: 4851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1284949, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:47,844 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 577ms (threshold=20ms); ack: seqno: 5018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1305067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:47,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 5068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 723843, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:47,909 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629805_1099518554435] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 9915 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 772540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:48,545 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 549ms (threshold=20ms); ack: seqno: 5185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1278927, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:48,694 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 5345 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 753367, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:48,694 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629805_1099518554435] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 14333 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 419879, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:48,695 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:48,695 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:48,695 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:48,695 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:48,695 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:49,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 547ms (threshold=20ms); ack: seqno: 5352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1344338, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:49,421 DEBUG [PriorityRpcServer.handler=9,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:43:49,421 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134275984 > lower bound=16777216 2015-10-16 12:43:49,421 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:43:49,449 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629805_1099518554435] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 18577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 639676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:49,548 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629806_1099518554436] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 78 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790823, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:43:49,548 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629805_1099518554435] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 18969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 442444, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:49,548 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:49,548 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:49,548 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:49,549 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:49,946 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 575ms (threshold=20ms); ack: seqno: 5521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1205279, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:50,161 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629806_1099518554436] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 931 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 734951, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:43:50,161 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629805_1099518554435] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 22193 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 473020, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:50,161 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:50,161 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:50,161 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:50,161 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:50,161 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:50,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2150517, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/10fd36b3642d434f827a4c1be2b320a0 2015-10-16 12:43:50,310 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/10fd36b3642d434f827a4c1be2b320a0 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/10fd36b3642d434f827a4c1be2b320a0 2015-10-16 12:43:50,316 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/10fd36b3642d434f827a4c1be2b320a0, entries=466310, sequenceid=2150517, filesize=70.8 M 2015-10-16 12:43:50,317 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134292560, currentsize=24.81 MB/26017280 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 896ms, sequenceid=2150517, compaction requested=true 2015-10-16 12:43:50,317 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:43:50,318 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 5 compacting, 3 eligible, 10 blocking 2015-10-16 12:43:50,319 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222841554 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 12:43:50,319 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:43:50,319 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:43:50,319 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 12:43:50,319 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/84cf48c60aeb42c594466cf9a26149f5, keycount=466280, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=2057246 2015-10-16 12:43:50,319 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ea89219e65e84607a0ad2cb5120b6c9a, keycount=466320, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=2103882 2015-10-16 12:43:50,320 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/10fd36b3642d434f827a4c1be2b320a0, keycount=466310, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=2150517 2015-10-16 12:43:50,335 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:43:50,589 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629805_1099518554435] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 23996 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 718033, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:50,589 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 109ms (threshold=20ms) 2015-10-16 12:43:50,589 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 109ms (threshold=20ms) 2015-10-16 12:43:50,589 INFO [sync.4] wal.FSHLog: Slow sync cost: 109 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:50,589 INFO [sync.3] wal.FSHLog: Slow sync cost: 109 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:50,589 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 109ms (threshold=20ms) 2015-10-16 12:43:50,589 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 109ms (threshold=20ms) 2015-10-16 12:43:50,589 INFO [sync.1] wal.FSHLog: Slow sync cost: 109 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:50,589 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 109ms (threshold=20ms) 2015-10-16 12:43:50,590 INFO [sync.0] wal.FSHLog: Slow sync cost: 109 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:50,590 INFO [sync.2] wal.FSHLog: Slow sync cost: 108 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:50,646 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 535ms (threshold=20ms); ack: seqno: 5688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1364331, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:50,998 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629805_1099518554435] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 26234 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 424438, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:50,998 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:50,998 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:50,998 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:50,998 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:51,126 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629805_1099518554435] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 26400 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 820525, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:51,740 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629807_1099518554437] hdfs.DFSClient: Slow ReadProcessor read fields took 680ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1345689, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:51,928 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:51,928 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629805_1099518554435] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 30773 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 377681, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:51,928 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:51,928 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:51,928 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:51,928 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:52,047 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 1276ms (threshold=20ms); ack: seqno: 5855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1422460, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:52,763 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629805_1099518554435] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 35601 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 496912, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:52,763 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:52,763 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:52,763 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:52,763 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:52,763 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:53,140 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629807_1099518554437] hdfs.DFSClient: Slow ReadProcessor read fields took 1288ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1398135, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:53,369 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:43:53,413 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024626072 with entries=77909, filesize=122.13 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024633369 2015-10-16 12:43:53,415 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024618280 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024618280 2015-10-16 12:43:53,416 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1036569, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:53,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 1270ms (threshold=20ms); ack: seqno: 6022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1046487, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:53,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 633 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 632992, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:53,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 6128 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 770184, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:53,566 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:53,566 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:53,566 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:53,566 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:53,566 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:53,940 DEBUG [PriorityRpcServer.handler=11,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:43:53,940 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134281632 > lower bound=16777216 2015-10-16 12:43:53,941 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.03 MB 2015-10-16 12:43:53,965 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 2765 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 549641, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:54,349 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:43:54,349 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629809_1099518554439] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 563 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839520, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:43:54,349 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 4792 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 324823, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:54,349 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:54,349 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:54,349 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:43:54,540 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629807_1099518554437] hdfs.DFSClient: Slow ReadProcessor read fields took 1278ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 976197, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:54,763 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2197152, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/93c4367f9d814f04a026304927694662 2015-10-16 12:43:54,769 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/93c4367f9d814f04a026304927694662 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/93c4367f9d814f04a026304927694662 2015-10-16 12:43:54,792 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/93c4367f9d814f04a026304927694662, entries=466310, sequenceid=2197152, filesize=70.8 M 2015-10-16 12:43:54,793 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134293600, currentsize=25.11 MB/26330880 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 853ms, sequenceid=2197152, compaction requested=false 2015-10-16 12:43:54,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629803_1099518554433] hdfs.DFSClient: Slow ReadProcessor read fields took 1232ms (threshold=20ms); ack: seqno: 6189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1122695, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:54,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 7656 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811859, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:55,015 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 7851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 416740, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:55,015 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 6342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4774792, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:55,016 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:43:55,016 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:43:55,016 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:43:55,622 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 11052 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 446664, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:55,622 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 79ms (threshold=20ms) 2015-10-16 12:43:55,622 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 79ms (threshold=20ms) 2015-10-16 12:43:55,622 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 79ms (threshold=20ms) 2015-10-16 12:43:55,622 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 79ms (threshold=20ms) 2015-10-16 12:43:55,622 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 79ms (threshold=20ms) 2015-10-16 12:43:55,940 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629807_1099518554437] hdfs.DFSClient: Slow ReadProcessor read fields took 1276ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1216532, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:55,998 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 13224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 539292, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:55,998 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629807_1099518554437] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 712 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728122, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:55,998 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:55,998 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:43:55,998 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:55,998 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:55,998 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:56,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 1228ms (threshold=20ms); ack: seqno: 6358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1478164, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:56,776 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 17619 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 452040, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:56,776 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:56,776 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:56,776 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:56,777 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:56,777 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:57,340 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629807_1099518554437] hdfs.DFSClient: Slow ReadProcessor read fields took 1257ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1559695, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:57,662 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 22528 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 444094, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:57,662 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:57,662 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:57,662 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:43:57,662 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:57,663 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 1294ms (threshold=20ms); ack: seqno: 6525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1322063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:58,311 DEBUG [PriorityRpcServer.handler=16,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:43:58,311 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134290496 > lower bound=16777216 2015-10-16 12:43:58,312 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.03 MB 2015-10-16 12:43:58,339 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 26278 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 598743, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:58,494 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 26962 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 426724, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:58,494 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629811_1099518554441] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 164 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 641432, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:43:58,494 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:58,494 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:58,494 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:58,494 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:58,494 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:43:58,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629807_1099518554437] hdfs.DFSClient: Slow ReadProcessor read fields took 1285ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3151779, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:43:59,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 1277ms (threshold=20ms); ack: seqno: 6694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1149958, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:59,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629811_1099518554441] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1032 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 939306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:43:59,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 30143 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 500218, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:43:59,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 6713 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794131, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:43:59,099 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:59,099 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:43:59,099 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:43:59,099 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:59,099 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:43:59,187 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2243795, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/debc38b5320c40daa0b03cfc3fbe0177 2015-10-16 12:43:59,194 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/debc38b5320c40daa0b03cfc3fbe0177 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/debc38b5320c40daa0b03cfc3fbe0177 2015-10-16 12:43:59,199 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/debc38b5320c40daa0b03cfc3fbe0177, entries=466390, sequenceid=2243795, filesize=70.9 M 2015-10-16 12:43:59,200 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.09 MB/134316000, currentsize=24.16 MB/25334320 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 889ms, sequenceid=2243795, compaction requested=true 2015-10-16 12:43:59,200 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:43:59,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 33476 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 731305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:00,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 34825 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 404249, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:00,048 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:44:00,048 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:44:00,048 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:44:00,048 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:44:00,048 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:44:00,141 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629807_1099518554437] hdfs.DFSClient: Slow ReadProcessor read fields took 1255ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1117614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:00,463 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 1252ms (threshold=20ms); ack: seqno: 6861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1599787, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:44:00,828 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629808_1099518554438] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 39030 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 554628, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:00,829 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:00,829 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:00,829 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:00,829 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:00,829 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:00,831 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:44:00,879 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024633369 with entries=77961, filesize=122.21 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024640831 2015-10-16 12:44:00,881 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024626072 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024626072 2015-10-16 12:44:00,882 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629812_1099518554442] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1169910, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:01,542 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629807_1099518554437] hdfs.DFSClient: Slow ReadProcessor read fields took 1268ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1371496, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:01,683 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629807_1099518554437] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 1492 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 663294, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:01,683 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629812_1099518554442] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 4737 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 380408, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:01,683 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:01,683 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:01,683 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:01,683 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:01,684 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:01,863 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 1274ms (threshold=20ms); ack: seqno: 7028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1272472, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:44:02,502 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629812_1099518554442] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 9612 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 417102, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:02,502 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:02,502 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:02,502 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:02,502 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:02,502 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:02,653 DEBUG [PriorityRpcServer.handler=0,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:44:02,653 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:44:02,654 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 12:44:02,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629807_1099518554437] hdfs.DFSClient: Slow ReadProcessor read fields took 1253ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1390185, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:03,263 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 1270ms (threshold=20ms); ack: seqno: 7195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1264543, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:44:03,299 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629812_1099518554442] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 14448 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 429751, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:03,299 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 7202 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013732, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:44:03,299 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:03,299 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:03,299 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:03,299 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:04,190 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629812_1099518554442] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 19214 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769017, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:04,240 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629812_1099518554442] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 19321 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 679079, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:04,240 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:04,240 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:04,240 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:04,240 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:04,240 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:04,343 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629807_1099518554437] hdfs.DFSClient: Slow ReadProcessor read fields took 1273ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1523426, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:04,663 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 1241ms (threshold=20ms); ack: seqno: 7362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1327179, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:44:04,969 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:04,969 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629812_1099518554442] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 23503 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 482503, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:04,969 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:44:04,969 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:04,969 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:44:04,969 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:05,743 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629807_1099518554437] hdfs.DFSClient: Slow ReadProcessor read fields took 1273ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1197883, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:05,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629812_1099518554442] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 28804 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 519527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:05,888 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:05,888 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:05,888 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:05,888 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:05,888 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:06,064 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 1270ms (threshold=20ms); ack: seqno: 7529 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1341868, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:44:06,243 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629807_1099518554437] hdfs.DFSClient: Slow ReadProcessor read fields took 500ms (threshold=20ms); ack: seqno: 1836 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 500249008, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:06,677 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629812_1099518554442] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 33333 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 459950, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:06,677 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:06,677 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:06,677 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:06,677 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:06,677 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:07,143 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629807_1099518554437] hdfs.DFSClient: Slow ReadProcessor read fields took 838ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1505317, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:07,465 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 1267ms (threshold=20ms); ack: seqno: 7696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1377404, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:44:07,505 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629812_1099518554442] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 38154 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 441575, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:07,505 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 7705 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038293, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:44:07,506 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:07,506 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:07,506 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:07,506 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:07,738 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:44:07,779 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024640831 with entries=77921, filesize=122.15 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024647738 2015-10-16 12:44:07,782 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629814_1099518554444] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054622, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:44:08,451 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629814_1099518554444] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 3353 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 716520, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:44:08,542 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629813_1099518554443] hdfs.DFSClient: Slow ReadProcessor read fields took 1271ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1245010, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:08,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 1244ms (threshold=20ms); ack: seqno: 7863 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1161445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:44:09,165 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629814_1099518554444] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 7517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 458610, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:44:09,165 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:09,165 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:09,165 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:09,165 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:09,166 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:09,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629813_1099518554443] hdfs.DFSClient: Slow ReadProcessor read fields took 1265ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1232724, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:10,047 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629813_1099518554443] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 2448 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 685518, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:10,047 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:10,047 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629814_1099518554444] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 12657 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 446980, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:44:10,048 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:10,048 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:10,048 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:10,049 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:10,265 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 1276ms (threshold=20ms); ack: seqno: 8030 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1217017, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:44:10,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 141ms (threshold=20ms); ack: seqno: 8031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 141647665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:44:10,853 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629814_1099518554444] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 17243 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 673894, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:44:10,853 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:11,342 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629813_1099518554443] hdfs.DFSClient: Slow ReadProcessor read fields took 1254ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1142754, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:11,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 1200ms (threshold=20ms); ack: seqno: 8199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 6687001, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:44:11,715 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629810_1099518554440] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 8224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 776301, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:44:11,715 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629814_1099518554444] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 22010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 372040, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:44:11,715 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:11,715 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:11,715 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:11,715 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:11,715 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:12,494 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629814_1099518554444] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 26107 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 597180, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:44:12,676 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:44:12,676 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:44:12,676 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629814_1099518554444] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 27024 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 413133, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:44:12,676 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:44:12,676 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:12,676 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:12,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629813_1099518554443] hdfs.DFSClient: Slow ReadProcessor read fields took 1286ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1385498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:12,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629814_1099518554444] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 27440 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 899766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:44:12,781 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:12,781 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:12,781 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:12,781 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:12,788 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:44:13,065 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629815_1099518554445] hdfs.DFSClient: Slow ReadProcessor read fields took 1237ms (threshold=20ms); ack: seqno: 8369 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1253214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:13,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629814_1099518554444] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 31252 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 550986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:44:13,442 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:13,442 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:13,442 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:13,442 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:14,142 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629813_1099518554443] hdfs.DFSClient: Slow ReadProcessor read fields took 1289ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1206612, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:14,290 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629813_1099518554443] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 2998 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 684723, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:14,290 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629814_1099518554444] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 36248 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 438534, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:44:14,290 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:14,290 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:14,290 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:14,291 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:14,465 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629815_1099518554445] hdfs.DFSClient: Slow ReadProcessor read fields took 1271ms (threshold=20ms); ack: seqno: 8535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1323141, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:14,837 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:44:14,887 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024647738 with entries=78010, filesize=122.29 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024654837 2015-10-16 12:44:14,891 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1199242, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:15,149 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 1395 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 534231, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:15,149 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:15,149 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:15,149 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:15,149 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:15,149 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:15,543 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629813_1099518554443] hdfs.DFSClient: Slow ReadProcessor read fields took 1243ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1547060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:15,866 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629815_1099518554445] hdfs.DFSClient: Slow ReadProcessor read fields took 1279ms (threshold=20ms); ack: seqno: 8702 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1470461, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:16,943 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629813_1099518554443] hdfs.DFSClient: Slow ReadProcessor read fields took 1290ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1361614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:17,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629815_1099518554445] hdfs.DFSClient: Slow ReadProcessor read fields took 1313ms (threshold=20ms); ack: seqno: 8871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 16727463, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:18,343 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629813_1099518554443] hdfs.DFSClient: Slow ReadProcessor read fields took 1299ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1492503, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:18,408 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#12 average throughput is 7.36 MB/sec, slept 20 time(s) and total slept time is 24388 ms. 1 active compactions remaining, total limit is 14.29 MB/sec 2015-10-16 12:44:18,428 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b8405f5c2b23400c8fb8f4536ecd6133 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b8405f5c2b23400c8fb8f4536ecd6133 2015-10-16 12:44:18,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 3120ms (threshold=20ms); ack: seqno: 2422 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825337, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:18,442 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:44:18,447 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:44:18,453 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/84cf48c60aeb42c594466cf9a26149f5, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/84cf48c60aeb42c594466cf9a26149f5 2015-10-16 12:44:18,461 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ea89219e65e84607a0ad2cb5120b6c9a, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ea89219e65e84607a0ad2cb5120b6c9a 2015-10-16 12:44:18,469 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/10fd36b3642d434f827a4c1be2b320a0, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/10fd36b3642d434f827a4c1be2b320a0 2015-10-16 12:44:18,470 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into b8405f5c2b23400c8fb8f4536ecd6133(size=212.5 M), total size for store is 3.3 G. This selection was in queue for 0sec, and took 28sec to execute. 2015-10-16 12:44:18,470 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=2, time=36902539173123473; duration=28sec 2015-10-16 12:44:18,470 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 12:44:18,470 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 5 compacting, 3 eligible, 10 blocking 2015-10-16 12:44:18,472 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:44:18,472 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:44:18,472 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:44:18,472 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 5 compacting, 3 eligible, 10 blocking 2015-10-16 12:44:18,474 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:44:18,474 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:44:18,474 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:44:18,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629815_1099518554445] hdfs.DFSClient: Slow ReadProcessor read fields took 1299ms (threshold=20ms); ack: seqno: 9038 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1643143, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:18,857 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536910096 > lower bound=16777216 2015-10-16 12:44:18,857 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=512.04 MB 2015-10-16 12:44:18,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 524ms (threshold=20ms); ack: seqno: 2423 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948497, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:19,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629817_1099518554447] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 240 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 686489, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:44:19,367 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629815_1099518554445] hdfs.DFSClient: Slow ReadProcessor read fields took 594ms (threshold=20ms); ack: seqno: 9205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1788373, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:19,377 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#11 average throughput is 12.86 MB/sec, slept 38 time(s) and total slept time is 35410 ms. 0 active compactions remaining, total limit is 14.29 MB/sec 2015-10-16 12:44:19,409 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629815_1099518554445] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 9221 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1183828, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:19,426 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ef434856016c4876908d88c5bcf538a0 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ef434856016c4876908d88c5bcf538a0 2015-10-16 12:44:19,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 476ms (threshold=20ms); ack: seqno: 2424 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 682897, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:19,443 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:44:19,448 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:44:19,453 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4beb8db3487d4195a34893d32ddbaed3, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4beb8db3487d4195a34893d32ddbaed3 2015-10-16 12:44:19,461 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/060a30edd42641ceb5cf26ada00bd31e, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/060a30edd42641ceb5cf26ada00bd31e 2015-10-16 12:44:19,469 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bbfda5027a904d2993e04ab79313663f, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bbfda5027a904d2993e04ab79313663f 2015-10-16 12:44:19,478 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/27bc39bdbe8c495eb24a465bb726f632, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/27bc39bdbe8c495eb24a465bb726f632 2015-10-16 12:44:19,486 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fdb2b87d945a423f8f16dd54d6226e1f, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fdb2b87d945a423f8f16dd54d6226e1f 2015-10-16 12:44:19,486 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into ef434856016c4876908d88c5bcf538a0(size=566.6 M), total size for store is 3.3 G. This selection was in queue for 0sec, and took 42sec to execute. 2015-10-16 12:44:19,486 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=5, fileSize=566.7 M, priority=4, time=36902525342529603; duration=42sec 2015-10-16 12:44:19,486 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:44:21,786 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2430233, memsize=512.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/491ecd32ed034f96bff3df4de0139d80 2015-10-16 12:44:21,794 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/491ecd32ed034f96bff3df4de0139d80 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/491ecd32ed034f96bff3df4de0139d80 2015-10-16 12:44:21,807 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/491ecd32ed034f96bff3df4de0139d80, entries=1864330, sequenceid=2430233, filesize=283.2 M 2015-10-16 12:44:21,808 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 2366ms (threshold=20ms); ack: seqno: 2425 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 600135, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:21,809 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~512.04 MB/536909680, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2952ms, sequenceid=2430233, compaction requested=true 2015-10-16 12:44:21,809 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:44:21,809 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 6 store files, 0 compacting, 6 eligible, 10 blocking 2015-10-16 12:44:21,812 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 5 files of size 1262538760 starting at candidate #1 after considering 10 permutations with 2 in ratio 2015-10-16 12:44:21,812 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:44:21,813 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:44:21,813 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=1.2 G 2015-10-16 12:44:21,813 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ef434856016c4876908d88c5bcf538a0, keycount=3730180, bloomtype=ROW, size=566.6 M, encoding=NONE, seqNum=2010614 2015-10-16 12:44:21,813 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b8405f5c2b23400c8fb8f4536ecd6133, keycount=1398910, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=2150517 2015-10-16 12:44:21,813 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/93c4367f9d814f04a026304927694662, keycount=466310, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=2197152 2015-10-16 12:44:21,813 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/debc38b5320c40daa0b03cfc3fbe0177, keycount=466390, bloomtype=ROW, size=70.9 M, encoding=NONE, seqNum=2243795 2015-10-16 12:44:21,813 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/491ecd32ed034f96bff3df4de0139d80, keycount=1864330, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=2430233 2015-10-16 12:44:21,843 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:44:22,556 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629820_1099518554450] hdfs.DFSClient: Slow ReadProcessor read fields took 597ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1500357, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:44:23,257 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629820_1099518554450] hdfs.DFSClient: Slow ReadProcessor read fields took 601ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1494770, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:44:23,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 1639ms (threshold=20ms); ack: seqno: 2426 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:23,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629820_1099518554450] hdfs.DFSClient: Slow ReadProcessor read fields took 600ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1413466, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:44:24,397 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:44:24,397 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 7872 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 417429, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:24,397 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:44:24,397 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:44:24,397 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:44:24,435 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 12:44:24,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 8222 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 641989, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:24,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629820_1099518554450] hdfs.DFSClient: Slow ReadProcessor read fields took 583ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1392969, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:44:25,319 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:25,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 12715 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 442811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:25,319 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:25,319 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:25,319 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:25,319 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:25,357 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629820_1099518554450] hdfs.DFSClient: Slow ReadProcessor read fields took 581ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1371937, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:44:26,058 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629820_1099518554450] hdfs.DFSClient: Slow ReadProcessor read fields took 581ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1282548, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:44:26,122 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:44:26,122 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:44:26,122 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:44:26,122 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:44:26,122 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629820_1099518554450] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 1050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729427, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:44:26,122 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:44:26,535 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 19610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 552413, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:26,758 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629820_1099518554450] hdfs.DFSClient: Slow ReadProcessor read fields took 549ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1574730, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:44:26,922 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 21832 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 447338, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:26,923 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:26,923 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:26,923 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:27,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629820_1099518554450] hdfs.DFSClient: Slow ReadProcessor read fields took 576ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1569200, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:44:27,601 DEBUG [PriorityRpcServer.handler=4,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:44:27,601 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134275616 > lower bound=16777216 2015-10-16 12:44:27,601 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:44:27,629 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 26037 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 463345, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:27,773 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629821_1099518554451] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 151 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797664, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:27,773 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 26717 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 551223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:27,773 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:27,773 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:27,773 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:27,773 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:27,773 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:28,064 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629820_1099518554450] hdfs.DFSClient: Slow ReadProcessor read fields took 605ms (threshold=20ms); ack: seqno: 1335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 604979292, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:44:28,159 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629820_1099518554450] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819472, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:44:28,261 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 28971 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 601489, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:28,366 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:44:28,366 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629821_1099518554451] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 1075 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 749708, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:28,366 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:44:28,366 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 29441 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 488590, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:28,366 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:44:28,366 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:44:28,366 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:44:28,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2476865, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/9fb55098112e414a9abb6e6548f62644 2015-10-16 12:44:28,424 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/9fb55098112e414a9abb6e6548f62644 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9fb55098112e414a9abb6e6548f62644 2015-10-16 12:44:28,431 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9fb55098112e414a9abb6e6548f62644, entries=466270, sequenceid=2476865, filesize=70.8 M 2015-10-16 12:44:28,432 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134280960, currentsize=20.81 MB/21818160 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 831ms, sequenceid=2476865, compaction requested=false 2015-10-16 12:44:28,860 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629820_1099518554450] hdfs.DFSClient: Slow ReadProcessor read fields took 590ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1496348, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:44:29,204 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 34134 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 424097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:29,204 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:44:29,204 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:44:29,204 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:44:29,204 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:44:29,204 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:44:29,561 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629820_1099518554450] hdfs.DFSClient: Slow ReadProcessor read fields took 548ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1349732, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:44:30,035 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629816_1099518554446] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 38969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 422543, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:30,035 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:30,035 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:30,035 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:30,035 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:30,074 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:44:30,103 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024654837 with entries=77792, filesize=121.94 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024670074 2015-10-16 12:44:30,104 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024633369 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024633369 2015-10-16 12:44:30,111 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024640831 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024640831 2015-10-16 12:44:30,119 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024647738 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024647738 2015-10-16 12:44:30,260 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629820_1099518554450] hdfs.DFSClient: Slow ReadProcessor read fields took 547ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1116922, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:44:30,653 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629822_1099518554452] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 2856 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 27241711, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:30,653 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:30,654 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:30,654 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:30,654 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:30,654 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:30,980 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629822_1099518554452] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 4525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 397377, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:30,980 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:30,980 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:30,980 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:30,982 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629823_1099518554453] hdfs.DFSClient: Slow ReadProcessor read fields took 576ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1359074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:31,682 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629823_1099518554453] hdfs.DFSClient: Slow ReadProcessor read fields took 546ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1413361, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:31,809 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629823_1099518554453] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 2441 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 739010, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:31,809 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629822_1099518554452] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 8990 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 648429, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:31,809 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:31,809 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:31,809 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:31,809 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:31,809 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:31,974 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629822_1099518554452] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 9388 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 754961, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:32,110 DEBUG [PriorityRpcServer.handler=3,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:44:32,110 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134270816 > lower bound=16777216 2015-10-16 12:44:32,111 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:44:32,126 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=659, evicted=0, evictedPerRun=0.0 2015-10-16 12:44:32,136 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629822_1099518554452] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 10097 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 748380, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:32,382 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629823_1099518554453] hdfs.DFSClient: Slow ReadProcessor read fields took 514ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1034850, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:32,588 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629822_1099518554452] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 12305 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 676929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:32,588 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629824_1099518554454] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 806219, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:32,588 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:32,588 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:32,588 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:32,588 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:32,588 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:32,944 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2523495, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/274da7b09bac478cb050d299e1025271 2015-10-16 12:44:32,950 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/274da7b09bac478cb050d299e1025271 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/274da7b09bac478cb050d299e1025271 2015-10-16 12:44:32,964 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/274da7b09bac478cb050d299e1025271, entries=466260, sequenceid=2523495, filesize=70.8 M 2015-10-16 12:44:32,966 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134279040, currentsize=23.68 MB/24827920 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 854ms, sequenceid=2523495, compaction requested=true 2015-10-16 12:44:32,966 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:44:32,966 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 5 compacting, 2 eligible, 10 blocking 2015-10-16 12:44:32,967 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:44:32,967 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:44:32,967 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:44:33,083 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629823_1099518554453] hdfs.DFSClient: Slow ReadProcessor read fields took 548ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1427588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:33,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629822_1099518554452] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 16173 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 406869, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:33,304 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:44:33,304 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:44:33,304 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:44:33,304 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:44:33,304 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:44:33,782 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629823_1099518554453] hdfs.DFSClient: Slow ReadProcessor read fields took 578ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1429997, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:34,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629822_1099518554452] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 21021 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 407551, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:34,216 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:34,216 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:34,216 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:34,216 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:34,216 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:34,482 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629823_1099518554453] hdfs.DFSClient: Slow ReadProcessor read fields took 570ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1452041, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:35,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629822_1099518554452] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 25963 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 616448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:35,129 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:35,129 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:35,129 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:35,129 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:35,129 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:35,182 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629823_1099518554453] hdfs.DFSClient: Slow ReadProcessor read fields took 569ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1523288, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:35,645 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629822_1099518554452] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 28321 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 554858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:35,882 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629823_1099518554453] hdfs.DFSClient: Slow ReadProcessor read fields took 573ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1380686, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:36,018 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:36,018 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:36,018 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:36,018 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:36,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629822_1099518554452] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 30250 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 429824, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:36,018 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:36,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629823_1099518554453] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 3481 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 875174, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:36,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629823_1099518554453] hdfs.DFSClient: Slow ReadProcessor read fields took 543ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1866486, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:36,617 DEBUG [PriorityRpcServer.handler=16,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:44:36,618 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134274080 > lower bound=16777216 2015-10-16 12:44:36,618 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:44:36,645 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629822_1099518554452] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 33523 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 635331, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:36,837 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629822_1099518554452] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 34434 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 574665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:36,837 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629825_1099518554455] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 243 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862439, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:44:36,837 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:36,837 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:36,837 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:36,837 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:36,837 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:37,297 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629823_1099518554453] hdfs.DFSClient: Slow ReadProcessor read fields took 590ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 16259144, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=103365, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 12:44:37,433 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629822_1099518554452] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 37570 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 451550, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:37,433 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629823_1099518554453] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 3829 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 721936, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:37,434 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:37,434 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:37,434 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:37,434 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:37,434 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:37,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2570124, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/02868f76595247959f2e4a28cb37b014 2015-10-16 12:44:37,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/02868f76595247959f2e4a28cb37b014 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/02868f76595247959f2e4a28cb37b014 2015-10-16 12:44:37,448 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/02868f76595247959f2e4a28cb37b014, entries=466250, sequenceid=2570124, filesize=70.8 M 2015-10-16 12:44:37,450 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134275680, currentsize=22.72 MB/23825360 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 832ms, sequenceid=2570124, compaction requested=true 2015-10-16 12:44:37,450 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:44:37,450 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 5 compacting, 3 eligible, 10 blocking 2015-10-16 12:44:37,452 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222821856 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 12:44:37,452 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:44:37,452 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:44:37,452 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 12:44:37,452 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9fb55098112e414a9abb6e6548f62644, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=2476865 2015-10-16 12:44:37,453 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/274da7b09bac478cb050d299e1025271, keycount=466260, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=2523495 2015-10-16 12:44:37,453 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/02868f76595247959f2e4a28cb37b014, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=2570124 2015-10-16 12:44:37,469 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:44:37,700 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:44:37,744 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024670074 with entries=77937, filesize=122.17 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024677700 2015-10-16 12:44:37,745 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024654837 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024654837 2015-10-16 12:44:37,983 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629823_1099518554453] hdfs.DFSClient: Slow ReadProcessor read fields took 537ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1261896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:38,213 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629827_1099518554457] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 2520 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 561163, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:38,213 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:44:38,213 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:44:38,213 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:44:38,213 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:44:38,213 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:44:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.8571428571428571, tune compaction throughput to 18.57 MB/sec 2015-10-16 12:44:38,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 1248ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1377924, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:39,087 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629827_1099518554457] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 7308 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 482170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:39,088 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:39,088 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:39,088 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:39,088 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:39,088 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:39,196 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629827_1099518554457] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 7381 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 602849, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:39,383 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629823_1099518554453] hdfs.DFSClient: Slow ReadProcessor read fields took 1267ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1303892, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:39,955 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 957ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1171165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:40,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629827_1099518554457] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 12020 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 580619, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:40,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 377 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841447, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:40,018 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:40,018 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:40,018 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:40,018 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:40,018 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:40,458 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 937ms (threshold=20ms); ack: seqno: 4181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1284176, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:40,621 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 534ms (threshold=20ms); ack: seqno: 476 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1166382, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:40,807 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629827_1099518554457] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 16405 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 506849, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:40,807 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:40,807 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:40,807 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:40,807 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:40,807 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:41,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 404ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207692, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:41,085 DEBUG [PriorityRpcServer.handler=17,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:44:41,085 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134270048 > lower bound=16777216 2015-10-16 12:44:41,085 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:44:41,110 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629827_1099518554457] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 17878 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 472292, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:41,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 947ms (threshold=20ms); ack: seqno: 4348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:41,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629827_1099518554457] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 20169 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 468077, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:41,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629829_1099518554459] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 548 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 791529, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:44:41,566 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:41,566 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:41,566 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:41,566 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:41,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 4352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1103528, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:41,566 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:42,035 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2616755, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/e8b2aa2780da48c29e124e248884f084 2015-10-16 12:44:42,040 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/e8b2aa2780da48c29e124e248884f084 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e8b2aa2780da48c29e124e248884f084 2015-10-16 12:44:42,047 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e8b2aa2780da48c29e124e248884f084, entries=466270, sequenceid=2616755, filesize=70.8 M 2015-10-16 12:44:42,048 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134281440, currentsize=27.21 MB/28534240 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 963ms, sequenceid=2616755, compaction requested=false 2015-10-16 12:44:42,107 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 955ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1222605, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:42,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629827_1099518554457] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 23550 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 572141, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:42,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 750 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 854024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:42,195 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:44:42,195 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:44:42,195 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:44:42,195 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:44:42,195 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:44:42,543 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629827_1099518554457] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 24934 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 722692, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:42,610 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 921ms (threshold=20ms); ack: seqno: 4517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1345368, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:43,122 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:44:43,122 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:44:43,122 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:44:43,122 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629827_1099518554457] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 28080 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 391010, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:43,122 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:44:43,122 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:44:43,183 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 931ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1108347, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:43,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 950ms (threshold=20ms); ack: seqno: 4684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1249994, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:43,897 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629827_1099518554457] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 32225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 592730, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:43,897 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:43,897 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:43,897 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:43,897 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:43,897 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:44,259 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 956ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188051, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:44,785 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629827_1099518554457] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 37074 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 376664, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:44:44,785 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:44,785 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:44:44,785 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:44:44,785 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:44:44,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 979ms (threshold=20ms); ack: seqno: 4851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1265708, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:45,126 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:44:45,182 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:44:45,193 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024677700 with entries=78255, filesize=122.67 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024685126 2015-10-16 12:44:45,195 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024670074 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024670074 2015-10-16 12:44:45,336 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 955ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1333667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:45,556 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629830_1099518554460] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 1945 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 483474, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:45,557 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:45,557 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:45,557 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:45,557 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:45,559 DEBUG [PriorityRpcServer.handler=2,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:44:45,559 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134276896 > lower bound=16777216 2015-10-16 12:44:45,560 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:44:45,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629830_1099518554460] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1970 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 497314, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:45,756 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629830_1099518554460] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 2417 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 598579, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:45,863 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 958ms (threshold=20ms); ack: seqno: 5018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1261514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:46,266 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629830_1099518554460] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 5008 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 608720, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:46,266 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629831_1099518554461] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 918 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 889026, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:46,411 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 955ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1130689, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:46,418 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2663384, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b9105be05ae14473a0ce9ed06a4ca49c 2015-10-16 12:44:46,424 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b9105be05ae14473a0ce9ed06a4ca49c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b9105be05ae14473a0ce9ed06a4ca49c 2015-10-16 12:44:46,430 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b9105be05ae14473a0ce9ed06a4ca49c, entries=466250, sequenceid=2663384, filesize=70.8 M 2015-10-16 12:44:46,431 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134276480, currentsize=21.78 MB/22842960 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 871ms, sequenceid=2663384, compaction requested=true 2015-10-16 12:44:46,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:44:46,939 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 5185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1235543, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:47,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629830_1099518554460] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 9146 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 575541, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:47,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 5242 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 780023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:47,017 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:47,017 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:47,017 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:47,017 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:47,017 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:47,488 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 964ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1372296, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:47,795 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:44:47,795 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:44:47,795 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:44:47,795 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:44:47,795 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:44:48,015 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 918ms (threshold=20ms); ack: seqno: 5352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1073646, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:48,563 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 963ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1369602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:48,624 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:48,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629830_1099518554460] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 18309 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 453152, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:48,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 1711 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 746974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:48,624 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:48,624 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:48,624 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:48,624 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:49,091 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 945ms (threshold=20ms); ack: seqno: 5521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1146961, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:49,377 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629830_1099518554460] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 22696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 619167, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:49,377 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:49,377 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:49,377 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:49,377 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:49,377 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:49,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629830_1099518554460] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 22814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769378, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:49,640 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 924ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1443826, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:49,935 DEBUG [PriorityRpcServer.handler=0,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:44:49,935 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:44:49,935 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 12:44:50,167 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 946ms (threshold=20ms); ack: seqno: 5688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1253575, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:50,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629830_1099518554460] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 27242 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 529198, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:50,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 5775 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 757550, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:50,258 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:50,258 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:50,258 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:50,258 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:50,258 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:44:50,716 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629826_1099518554456] hdfs.DFSClient: Slow ReadProcessor read fields took 954ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1423060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:50,999 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:44:50,999 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:44:50,999 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:44:50,999 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:50,999 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:44:51,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 921ms (threshold=20ms); ack: seqno: 5855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1338836, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:51,821 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629830_1099518554460] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 36628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 581663, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:44:51,821 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629834_1099518554464] hdfs.DFSClient: Slow ReadProcessor read fields took 980ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1175508, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:51,821 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:51,821 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:51,821 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:51,821 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:51,821 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:52,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:44:52,310 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024685126 with entries=77869, filesize=122.07 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024692273 2015-10-16 12:44:52,311 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024677700 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024677700 2015-10-16 12:44:52,320 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 946ms (threshold=20ms); ack: seqno: 6022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1309141, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:52,573 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629835_1099518554465] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 1358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 475461, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:52,574 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:52,574 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:52,574 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:52,867 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629834_1099518554464] hdfs.DFSClient: Slow ReadProcessor read fields took 916ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1528544, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:53,397 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 947ms (threshold=20ms); ack: seqno: 6189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1526789, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:53,451 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629835_1099518554465] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 6132 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 503754, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:53,451 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629828_1099518554458] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 6216 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835704, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:44:53,451 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:53,451 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:53,451 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:53,905 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629835_1099518554465] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 8203 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:53,944 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629834_1099518554464] hdfs.DFSClient: Slow ReadProcessor read fields took 948ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1448388, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:54,317 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629835_1099518554465] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 10422 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 562642, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:54,318 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:54,318 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:54,318 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:54,318 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:44:54,472 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 915ms (threshold=20ms); ack: seqno: 6358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1075170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:55,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629834_1099518554464] hdfs.DFSClient: Slow ReadProcessor read fields took 948ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1413425, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:55,125 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629835_1099518554465] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 14893 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 507098, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:55,125 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629834_1099518554464] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 2771 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 843522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:55,125 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:55,125 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:55,125 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:55,125 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:55,125 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:55,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629835_1099518554465] hdfs.DFSClient: Slow ReadProcessor read fields took 198ms (threshold=20ms); ack: seqno: 15116 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787765, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:55,360 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 198ms (threshold=20ms) 2015-10-16 12:44:55,360 INFO [sync.1] wal.FSHLog: Slow sync cost: 198 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:55,360 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 198ms (threshold=20ms) 2015-10-16 12:44:55,360 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 198ms (threshold=20ms) 2015-10-16 12:44:55,361 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 198ms (threshold=20ms) 2015-10-16 12:44:55,361 INFO [sync.4] wal.FSHLog: Slow sync cost: 198 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:55,361 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 198ms (threshold=20ms) 2015-10-16 12:44:55,361 INFO [sync.2] wal.FSHLog: Slow sync cost: 198 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:55,361 INFO [sync.3] wal.FSHLog: Slow sync cost: 198 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:55,361 INFO [sync.0] wal.FSHLog: Slow sync cost: 198 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:55,547 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 943ms (threshold=20ms); ack: seqno: 6525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017046, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:56,095 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629834_1099518554464] hdfs.DFSClient: Slow ReadProcessor read fields took 920ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1520867, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:56,142 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629834_1099518554464] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 2865 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 799806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:56,142 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629835_1099518554465] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 19394 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 420201, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:56,142 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:56,142 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:56,142 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:56,142 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:44:56,142 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:56,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 936ms (threshold=20ms); ack: seqno: 6694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 998957, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:56,919 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629835_1099518554465] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 23539 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 608263, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:56,919 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:44:56,919 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:44:56,919 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:44:56,919 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:44:56,919 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:44:56,952 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629835_1099518554465] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 23552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 620805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:57,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629834_1099518554464] hdfs.DFSClient: Slow ReadProcessor read fields took 918ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1593138, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:57,700 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 947ms (threshold=20ms); ack: seqno: 6861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054979, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:57,786 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629835_1099518554465] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 28131 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 468396, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:57,786 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 6937 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 695204, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:57,786 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:57,786 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:44:57,786 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:57,786 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:57,786 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:44:58,247 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629834_1099518554464] hdfs.DFSClient: Slow ReadProcessor read fields took 949ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1392064, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:58,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629835_1099518554465] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 31988 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847356, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:58,717 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629835_1099518554465] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 32653 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 503266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:58,717 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:44:58,717 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:44:58,717 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:44:58,718 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:44:58,718 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:44:58,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 942ms (threshold=20ms); ack: seqno: 7028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 19533371, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:59,323 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629834_1099518554464] hdfs.DFSClient: Slow ReadProcessor read fields took 953ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1371373, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:44:59,407 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#14 average throughput is 9.42 MB/sec, slept 20 time(s) and total slept time is 19159 ms. 1 active compactions remaining, total limit is 18.57 MB/sec 2015-10-16 12:44:59,431 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/23039f7f013e4bc196eb3bcc4c19658e as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/23039f7f013e4bc196eb3bcc4c19658e 2015-10-16 12:44:59,439 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:44:59,442 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:44:59,450 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9fb55098112e414a9abb6e6548f62644, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9fb55098112e414a9abb6e6548f62644 2015-10-16 12:44:59,458 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/274da7b09bac478cb050d299e1025271, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/274da7b09bac478cb050d299e1025271 2015-10-16 12:44:59,467 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/02868f76595247959f2e4a28cb37b014, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/02868f76595247959f2e4a28cb37b014 2015-10-16 12:44:59,467 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 23039f7f013e4bc196eb3bcc4c19658e(size=212.5 M), total size for store is 4.0 G. This selection was in queue for 0sec, and took 22sec to execute. 2015-10-16 12:44:59,467 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=2, time=36902586305909978; duration=22sec 2015-10-16 12:44:59,467 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 12:44:59,467 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 5 compacting, 3 eligible, 10 blocking 2015-10-16 12:44:59,469 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:44:59,469 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:44:59,469 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:44:59,469 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 5 compacting, 3 eligible, 10 blocking 2015-10-16 12:44:59,471 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:44:59,471 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:44:59,471 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:44:59,575 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629835_1099518554465] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 37080 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 591088, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:44:59,575 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:59,575 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:59,576 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:44:59,576 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:59,576 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:44:59,837 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=432430736 > lower bound=16777216 2015-10-16 12:44:59,837 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=412.34 MB 2015-10-16 12:44:59,852 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 954ms (threshold=20ms); ack: seqno: 7195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1136139, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:44:59,921 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629835_1099518554465] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 38468 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 706378, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:45:00,064 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:45:00,117 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024692273 with entries=77966, filesize=122.22 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024700064 2015-10-16 12:45:00,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629838_1099518554468] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988289, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:00,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629837_1099518554467] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 635 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870505, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:45:00,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629838_1099518554468] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 1352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 518783, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:00,407 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:45:00,407 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:45:00,407 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:45:00,407 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:45:00,407 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:45:00,408 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 444ms (threshold=20ms); ack: seqno: 7362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909675, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:45:00,947 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 419ms (threshold=20ms); ack: seqno: 7529 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:45:01,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629837_1099518554467] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 1532 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797077, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:45:01,026 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629838_1099518554468] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 4380 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 557847, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:01,026 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:45:01,026 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:45:01,026 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 7577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 828695, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:45:01,026 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:45:01,026 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:45:01,485 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 371ms (threshold=20ms); ack: seqno: 7696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1175363, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:45:02,024 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 392ms (threshold=20ms); ack: seqno: 7863 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1177016, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:45:02,157 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:45:02,157 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:45:02,157 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:45:02,157 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 8013 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 770307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:45:02,157 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:45:02,467 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2813543, memsize=412.4 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b37c2872d2f44aed879c71720fc1cf38 2015-10-16 12:45:02,473 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b37c2872d2f44aed879c71720fc1cf38 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b37c2872d2f44aed879c71720fc1cf38 2015-10-16 12:45:02,478 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b37c2872d2f44aed879c71720fc1cf38, entries=1501540, sequenceid=2813543, filesize=228.1 M 2015-10-16 12:45:02,480 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~412.40 MB/432430320, currentsize=72.30 MB/75808080 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2643ms, sequenceid=2813543, compaction requested=true 2015-10-16 12:45:02,480 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:45:02,480 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 5 compacting, 4 eligible, 10 blocking 2015-10-16 12:45:02,482 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 610550793 starting at candidate #0 after considering 3 permutations with 1 in ratio 2015-10-16 12:45:02,482 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:45:02,482 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:45:02,482 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=582.3 M 2015-10-16 12:45:02,482 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/23039f7f013e4bc196eb3bcc4c19658e, keycount=1398780, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=2570124 2015-10-16 12:45:02,482 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e8b2aa2780da48c29e124e248884f084, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=2616755 2015-10-16 12:45:02,483 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b9105be05ae14473a0ce9ed06a4ca49c, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=2663384 2015-10-16 12:45:02,483 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b37c2872d2f44aed879c71720fc1cf38, keycount=1501540, bloomtype=ROW, size=228.1 M, encoding=NONE, seqNum=2813543 2015-10-16 12:45:02,505 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:45:02,562 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 392ms (threshold=20ms); ack: seqno: 8030 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956468, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:45:02,744 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:02,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629838_1099518554468] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 13589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 429494, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:02,744 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:02,744 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:02,744 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:02,745 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:45:03,080 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629838_1099518554468] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 15092 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 577356, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:03,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 945ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1195443, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:03,637 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 940ms (threshold=20ms); ack: seqno: 8199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 968249, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:45:03,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 272 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 871300, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:03,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 8242 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942899, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:45:03,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629838_1099518554468] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 18534 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 426976, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:03,699 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:45:03,699 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:45:03,700 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:45:04,260 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629836_1099518554466] hdfs.DFSClient: Slow ReadProcessor read fields took 491ms (threshold=20ms); ack: seqno: 8334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 490781978, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:45:04,371 DEBUG [PriorityRpcServer.handler=15,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:45:04,371 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134274448 > lower bound=16777216 2015-10-16 12:45:04,371 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:45:04,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629838_1099518554468] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 22453 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 424795, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:04,541 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629838_1099518554468] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 23040 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 479212, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:04,541 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629842_1099518554472] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 147 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 755379, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:04,541 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:04,541 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:04,541 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:04,663 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 914ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1059568, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:04,713 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 427ms (threshold=20ms); ack: seqno: 8368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1418814, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:05,088 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629838_1099518554468] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 25827 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 430717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:05,088 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629842_1099518554472] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 869 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758189, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:05,089 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:45:05,089 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:45:05,089 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:45:05,130 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629838_1099518554468] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 25831 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 40979382, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:05,130 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:45:05,130 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:45:05,130 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:45:05,130 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:45:05,130 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:45:05,283 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2860174, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d17768c1387840fc904553dd4e5b8395 2015-10-16 12:45:05,290 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d17768c1387840fc904553dd4e5b8395 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d17768c1387840fc904553dd4e5b8395 2015-10-16 12:45:05,295 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d17768c1387840fc904553dd4e5b8395, entries=466270, sequenceid=2860174, filesize=70.8 M 2015-10-16 12:45:05,296 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134281520, currentsize=23.65 MB/24798960 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 925ms, sequenceid=2860174, compaction requested=false 2015-10-16 12:45:05,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 944ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1268389, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:05,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 920ms (threshold=20ms); ack: seqno: 8535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1291231, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:05,874 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 645 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 771005, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:05,874 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629838_1099518554468] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 29891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 550190, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:05,874 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 8617 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 816003, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:05,874 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:45:05,874 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:45:05,874 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:45:05,874 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:45:05,874 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:45:06,800 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:45:06,801 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629838_1099518554468] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 35083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 414267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:06,801 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:45:06,810 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:45:06,810 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:45:06,810 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:45:06,815 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 923ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1340144, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:06,866 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 930ms (threshold=20ms); ack: seqno: 8702 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088838, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:07,198 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629838_1099518554468] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 36699 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 520660, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:07,592 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:45:07,633 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024700064 with entries=77948, filesize=122.19 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024707592 2015-10-16 12:45:07,635 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024685126 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024685126 2015-10-16 12:45:07,638 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629843_1099518554473] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2702919, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:07,642 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024692273 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024692273 2015-10-16 12:45:07,703 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629843_1099518554473] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 162 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 530261, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:07,703 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:45:07,704 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:45:07,704 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:45:07,704 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:45:07,704 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:45:07,891 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 943ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1271526, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:07,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 946ms (threshold=20ms); ack: seqno: 8871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1078757, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:08,516 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629843_1099518554473] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 4431 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 599595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:08,516 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:45:08,516 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:45:08,516 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:45:08,516 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:45:08,516 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:45:08,940 DEBUG [PriorityRpcServer.handler=2,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:45:08,940 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:45:08,940 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:45:08,968 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 943ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1165412, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:09,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 951ms (threshold=20ms); ack: seqno: 9038 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995901, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:09,297 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:09,297 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629843_1099518554473] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 8559 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 383990, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:09,297 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:09,297 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:10,044 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 937ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1305857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:10,094 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 945ms (threshold=20ms); ack: seqno: 9205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1106290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:10,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629843_1099518554473] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 12991 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 21752128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:10,111 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:45:10,111 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:45:10,111 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:45:10,111 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:45:10,111 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:45:10,180 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 9273 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922628, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:10,180 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629843_1099518554473] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 13205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 590930, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:10,180 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 1315 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922777, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:10,180 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:10,180 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:10,180 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:10,180 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:10,180 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:11,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 926ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1195358, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:11,157 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 1342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013759, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:11,157 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629843_1099518554473] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 18226 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 549872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:11,158 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:45:11,158 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:11,158 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:45:11,158 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:45:11,158 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:45:11,170 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 919ms (threshold=20ms); ack: seqno: 9372 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1042286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:11,264 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629843_1099518554473] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 18276 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769689, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:12,073 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629843_1099518554473] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 22489 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 658528, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:12,073 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:12,073 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:12,073 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:12,073 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:12,073 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:12,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 933ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1388252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:12,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 957ms (threshold=20ms); ack: seqno: 9539 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1192325, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:12,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629843_1099518554473] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 25719 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 765096, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:12,974 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629843_1099518554473] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 26773 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 662912, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:12,974 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:12,974 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:12,974 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:12,974 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:12,974 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:13,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 956ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1415995, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:13,322 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 944ms (threshold=20ms); ack: seqno: 9706 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1186326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:13,848 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629843_1099518554473] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 31017 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 748279, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:13,849 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:13,849 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:13,849 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:13,849 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:13,849 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:14,347 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 954ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1422040, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:14,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 953ms (threshold=20ms); ack: seqno: 9873 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988345, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:14,682 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629843_1099518554473] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 35160 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 390189, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:14,682 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:14,682 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:14,682 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:14,682 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:14,682 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:15,311 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629843_1099518554473] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 38133 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 629540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:15,424 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629840_1099518554470] hdfs.DFSClient: Slow ReadProcessor read fields took 954ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1506523, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:15,475 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 952ms (threshold=20ms); ack: seqno: 10040 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:15,486 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:45:15,541 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024707592 with entries=77947, filesize=122.19 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024715486 2015-10-16 12:45:15,546 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972506, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:15,616 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 204 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 676810, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:15,616 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 10186 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 763734, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:15,616 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:45:15,616 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:15,616 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:45:15,616 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:45:15,617 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:45:16,498 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629844_1099518554474] hdfs.DFSClient: Slow ReadProcessor read fields took 957ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1218123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:16,593 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 5352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 476176, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:16,593 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629844_1099518554474] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 2265 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811591, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:16,593 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:45:16,593 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:45:16,593 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:45:16,593 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:45:16,593 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:45:16,894 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 1264ms (threshold=20ms); ack: seqno: 10207 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 343853363, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:17,470 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 9991 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 505807, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:17,470 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:17,470 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:17,470 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:17,470 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:17,470 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:17,575 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629844_1099518554474] hdfs.DFSClient: Slow ReadProcessor read fields took 929ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1097789, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:17,626 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629841_1099518554471] hdfs.DFSClient: Slow ReadProcessor read fields took 663ms (threshold=20ms); ack: seqno: 10373 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1228717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:45:18,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 14218 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 453885, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:18,258 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:18,258 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:18,258 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:18,258 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:18,258 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:18,652 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629844_1099518554474] hdfs.DFSClient: Slow ReadProcessor read fields took 955ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1211495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:18,703 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 944ms (threshold=20ms); ack: seqno: 10543 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1161687, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:19,034 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 17928 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 724795, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:19,146 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 18390 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 543805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:19,146 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:45:19,146 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:45:19,146 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:45:19,146 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:19,146 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:19,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629844_1099518554474] hdfs.DFSClient: Slow ReadProcessor read fields took 953ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1202897, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:19,779 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 942ms (threshold=20ms); ack: seqno: 10710 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072617, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:19,935 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 10866 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 773319, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:19,935 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 22613 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701316, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:19,936 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:20,804 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629844_1099518554474] hdfs.DFSClient: Slow ReadProcessor read fields took 954ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1169656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:20,855 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 910ms (threshold=20ms); ack: seqno: 10879 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1244163, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:20,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629844_1099518554474] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 2918 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 867044, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:20,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 27891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 368914, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:20,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 10880 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 6790473, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:20,887 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:20,887 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:20,887 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:20,887 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:45:21,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 32444 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 529866, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:21,735 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:45:21,735 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:45:21,735 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:45:21,880 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629844_1099518554474] hdfs.DFSClient: Slow ReadProcessor read fields took 919ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054999, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:21,930 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#13 2462468/3832840 (64.25%), rate=11249.85 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=18.57 MB/sec, activeCompactions=2] 2015-10-16 12:45:21,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 915ms (threshold=20ms); ack: seqno: 11046 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1231532, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:22,056 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#13 sleep 949 ms because current throughput is 78.74 MB/sec, max allowed is 9.29 MB/sec, already slept 66 time(s) and total slept time is 50048 ms till now. 2015-10-16 12:45:22,558 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 36353 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 699360, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:22,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 36697 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 545699, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:22,647 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:22,647 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:22,647 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:22,647 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:22,647 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:45:22,956 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629844_1099518554474] hdfs.DFSClient: Slow ReadProcessor read fields took 952ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1138459, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:23,008 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 950ms (threshold=20ms); ack: seqno: 11215 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1231171, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:24,032 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629844_1099518554474] hdfs.DFSClient: Slow ReadProcessor read fields took 966ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1419773, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:24,085 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 969ms (threshold=20ms); ack: seqno: 11382 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1268844, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:25,109 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629844_1099518554474] hdfs.DFSClient: Slow ReadProcessor read fields took 965ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1419306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:25,160 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 970ms (threshold=20ms); ack: seqno: 11549 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1208448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:25,260 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 11657 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 704320, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:26,189 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629844_1099518554474] hdfs.DFSClient: Slow ReadProcessor read fields took 975ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3525805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:26,237 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 939ms (threshold=20ms); ack: seqno: 11716 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1296903, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:27,262 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629844_1099518554474] hdfs.DFSClient: Slow ReadProcessor read fields took 969ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1920390, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:27,312 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 970ms (threshold=20ms); ack: seqno: 11883 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1150395, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:28,337 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629844_1099518554474] hdfs.DFSClient: Slow ReadProcessor read fields took 973ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1525427, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:28,389 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 970ms (threshold=20ms); ack: seqno: 12050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1084599, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:29,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629847_1099518554477] hdfs.DFSClient: Slow ReadProcessor read fields took 945ms (threshold=20ms); ack: seqno: 4181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1801806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:29,465 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 971ms (threshold=20ms); ack: seqno: 12217 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1294201, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:29,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 12334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:30,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629847_1099518554477] hdfs.DFSClient: Slow ReadProcessor read fields took 972ms (threshold=20ms); ack: seqno: 4348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1680107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:30,542 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629846_1099518554476] hdfs.DFSClient: Slow ReadProcessor read fields took 947ms (threshold=20ms); ack: seqno: 12384 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1312828, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:31,564 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629847_1099518554477] hdfs.DFSClient: Slow ReadProcessor read fields took 971ms (threshold=20ms); ack: seqno: 4517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1700509, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:31,618 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629848_1099518554478] hdfs.DFSClient: Slow ReadProcessor read fields took 962ms (threshold=20ms); ack: seqno: 12553 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1539431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:45:32,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629847_1099518554477] hdfs.DFSClient: Slow ReadProcessor read fields took 974ms (threshold=20ms); ack: seqno: 4684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1632658, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:32,694 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629848_1099518554478] hdfs.DFSClient: Slow ReadProcessor read fields took 972ms (threshold=20ms); ack: seqno: 12720 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1117310, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:45:33,717 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629847_1099518554477] hdfs.DFSClient: Slow ReadProcessor read fields took 974ms (threshold=20ms); ack: seqno: 4851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1650445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:33,770 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629848_1099518554478] hdfs.DFSClient: Slow ReadProcessor read fields took 971ms (threshold=20ms); ack: seqno: 12887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1403509, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:45:33,871 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629848_1099518554478] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 13005 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 662267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:45:34,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629847_1099518554477] hdfs.DFSClient: Slow ReadProcessor read fields took 972ms (threshold=20ms); ack: seqno: 5018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1637958, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:34,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629848_1099518554478] hdfs.DFSClient: Slow ReadProcessor read fields took 941ms (threshold=20ms); ack: seqno: 13054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1361798, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:45:35,869 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629847_1099518554477] hdfs.DFSClient: Slow ReadProcessor read fields took 968ms (threshold=20ms); ack: seqno: 5185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1751405, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:35,923 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629848_1099518554478] hdfs.DFSClient: Slow ReadProcessor read fields took 970ms (threshold=20ms); ack: seqno: 13221 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1178434, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:45:36,081 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629848_1099518554478] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 13373 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 694455, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:45:36,946 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629847_1099518554477] hdfs.DFSClient: Slow ReadProcessor read fields took 965ms (threshold=20ms); ack: seqno: 5352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1790967, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:36,999 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629848_1099518554478] hdfs.DFSClient: Slow ReadProcessor read fields took 906ms (threshold=20ms); ack: seqno: 13390 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1370670, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:45:38,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629847_1099518554477] hdfs.DFSClient: Slow ReadProcessor read fields took 959ms (threshold=20ms); ack: seqno: 5521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1814046, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:38,077 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629848_1099518554478] hdfs.DFSClient: Slow ReadProcessor read fields took 964ms (threshold=20ms); ack: seqno: 13559 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1169893, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:45:38,185 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629848_1099518554478] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 13688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 745139, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:45:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 12:45:39,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629847_1099518554477] hdfs.DFSClient: Slow ReadProcessor read fields took 973ms (threshold=20ms); ack: seqno: 5688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1741177, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:39,154 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629848_1099518554478] hdfs.DFSClient: Slow ReadProcessor read fields took 942ms (threshold=20ms); ack: seqno: 13726 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1251819, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:45:39,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629849_1099518554479] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 6357 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2612448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:45:39,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629848_1099518554478] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 14359 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 765475, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:45:40,324 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629849_1099518554479] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 7401 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 642962, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:45:40,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629850_1099518554480] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 15185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1056683, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:40,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629849_1099518554479] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 8232 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808727, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:45:41,342 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629851_1099518554481] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 8886 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 724850, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:45:41,342 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629850_1099518554480] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 15875 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 677081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:41,766 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#15 average throughput is 14.42 MB/sec, slept 34 time(s) and total slept time is 32392 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 12:45:41,788 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/0afb955b872d42bbbf70f697b0ef7147 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0afb955b872d42bbbf70f697b0ef7147 2015-10-16 12:45:41,803 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 18915ms (threshold=20ms); ack: seqno: 38118 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 589236, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:41,803 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:45:41,808 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:45:41,829 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/23039f7f013e4bc196eb3bcc4c19658e, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/23039f7f013e4bc196eb3bcc4c19658e 2015-10-16 12:45:41,839 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e8b2aa2780da48c29e124e248884f084, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e8b2aa2780da48c29e124e248884f084 2015-10-16 12:45:41,847 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b9105be05ae14473a0ce9ed06a4ca49c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b9105be05ae14473a0ce9ed06a4ca49c 2015-10-16 12:45:41,856 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b37c2872d2f44aed879c71720fc1cf38, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b37c2872d2f44aed879c71720fc1cf38 2015-10-16 12:45:41,856 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 0afb955b872d42bbbf70f697b0ef7147(size=582.3 M), total size for store is 4.2 G. This selection was in queue for 0sec, and took 39sec to execute. 2015-10-16 12:45:41,856 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=582.3 M, priority=0, time=36902611335996643; duration=39sec 2015-10-16 12:45:41,856 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:45:41,856 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 5 compacting, 2 eligible, 10 blocking 2015-10-16 12:45:41,857 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:45:41,858 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:45:41,858 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:45:42,247 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536916816 > lower bound=16777216 2015-10-16 12:45:42,247 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=512.04 MB 2015-10-16 12:45:42,357 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 553ms (threshold=20ms); ack: seqno: 38119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838566, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:42,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629852_1099518554482] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 17785 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 667096, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:45:42,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629853_1099518554483] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 387 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 673286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:45:43,932 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#13 average throughput is 14.26 MB/sec, slept 82 time(s) and total slept time is 65345 ms. 0 active compactions remaining, total limit is unlimited 2015-10-16 12:45:43,954 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1620b55306cf4c398136d5c37f71317d as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1620b55306cf4c398136d5c37f71317d 2015-10-16 12:45:43,971 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 1614ms (threshold=20ms); ack: seqno: 38120 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 497169, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:43,971 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:45:43,972 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:45:43,981 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ef434856016c4876908d88c5bcf538a0, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ef434856016c4876908d88c5bcf538a0 2015-10-16 12:45:43,989 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b8405f5c2b23400c8fb8f4536ecd6133, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b8405f5c2b23400c8fb8f4536ecd6133 2015-10-16 12:45:43,997 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/93c4367f9d814f04a026304927694662, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/93c4367f9d814f04a026304927694662 2015-10-16 12:45:44,006 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/debc38b5320c40daa0b03cfc3fbe0177, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/debc38b5320c40daa0b03cfc3fbe0177 2015-10-16 12:45:44,014 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/491ecd32ed034f96bff3df4de0139d80, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/491ecd32ed034f96bff3df4de0139d80 2015-10-16 12:45:44,014 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 1620b55306cf4c398136d5c37f71317d(size=1.2 G), total size for store is 4.2 G. This selection was in queue for 0sec, and took 1mins, 22sec to execute. 2015-10-16 12:45:44,014 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=5, fileSize=1.2 G, priority=4, time=36902570666574897; duration=1mins, 22sec 2015-10-16 12:45:44,014 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:45:45,139 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3046614, memsize=512.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5766b7936c554fc5940d1b911ddb45c2 2015-10-16 12:45:45,145 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5766b7936c554fc5940d1b911ddb45c2 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5766b7936c554fc5940d1b911ddb45c2 2015-10-16 12:45:45,151 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5766b7936c554fc5940d1b911ddb45c2, entries=1864350, sequenceid=3046614, filesize=283.2 M 2015-10-16 12:45:45,153 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 1181ms (threshold=20ms); ack: seqno: 38121 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:45,153 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~512.04 MB/536916400, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2906ms, sequenceid=3046614, compaction requested=true 2015-10-16 12:45:45,153 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:45:45,153 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 5 store files, 0 compacting, 5 eligible, 10 blocking 2015-10-16 12:45:45,156 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 5 files of size 4851654908 starting at candidate #0 after considering 6 permutations with 1 in ratio 2015-10-16 12:45:45,156 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction (all files) 2015-10-16 12:45:45,156 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 5 store files, 0 compacting, 5 eligible, 10 blocking 2015-10-16 12:45:45,159 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 5 files of size 4851654908 starting at candidate #0 after considering 6 permutations with 1 in ratio 2015-10-16 12:45:45,159 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction (all files) 2015-10-16 12:45:45,159 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:45:45,160 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=4.5 G 2015-10-16 12:45:45,160 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/53d2a7c0edf3492cadeec7d370728efb, keycount=16374480, bloomtype=ROW, size=2.4 G, encoding=NONE, seqNum=1637574, earliestPutTs=1445024396631 2015-10-16 12:45:45,161 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1620b55306cf4c398136d5c37f71317d, keycount=7926120, bloomtype=ROW, size=1.2 G, encoding=NONE, seqNum=2430233, earliestPutTs=1445024569958 2015-10-16 12:45:45,161 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0afb955b872d42bbbf70f697b0ef7147, keycount=3832840, bloomtype=ROW, size=582.3 M, encoding=NONE, seqNum=2813543, earliestPutTs=1445024663445 2015-10-16 12:45:45,162 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d17768c1387840fc904553dd4e5b8395, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=2860174, earliestPutTs=1445024699840 2015-10-16 12:45:45,163 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5766b7936c554fc5940d1b911ddb45c2, keycount=1864350, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=3046614, earliestPutTs=1445024704371 2015-10-16 12:45:45,193 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:45:46,224 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629857_1099518554487] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 1627 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 752830, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:45:47,064 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629858_1099518554488] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 2980 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 743928, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:45:47,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629859_1099518554489] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 4338 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 695730, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:48,790 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629859_1099518554489] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 5687 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 726614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:45:49,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629860_1099518554490] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 7043 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 935561, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:50,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629861_1099518554491] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 8368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3312081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:51,057 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 5904ms (threshold=20ms); ack: seqno: 38122 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:51,103 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 38123 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701048, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:51,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 141ms (threshold=20ms); ack: seqno: 38125 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 629382, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:51,289 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 38127 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 613973, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:51,395 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629861_1099518554491] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 9726 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 689368, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:51,492 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 198ms (threshold=20ms); ack: seqno: 38145 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 626325, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:51,550 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 38149 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 771323, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:51,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 38159 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 654476, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:51,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 38168 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 625943, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:51,727 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 38170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813622, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:51,788 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 38180 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 628267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:51,896 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 38183 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 690273, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:51,996 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 38184 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 855067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:52,046 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 38186 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 576787, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:52,096 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 38193 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741289, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:52,143 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 38196 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:52,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 38208 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 666368, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:52,292 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 38216 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 630686, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:52,386 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 38228 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 554145, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:52,486 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 38239 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 581733, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:52,594 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 38254 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 571209, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:52,665 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629862_1099518554492] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 11050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 756530, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:52,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 150ms (threshold=20ms); ack: seqno: 38255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701538, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:52,796 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 38259 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 607755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:52,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 38272 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 670758, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:52,940 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 38276 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 578154, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:53,039 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 38295 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779601, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:53,089 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 38307 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 447344, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:53,181 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 38308 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 561119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:53,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 38329 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 481831, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:53,376 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 38342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 519727, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:53,427 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 38372 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 457195, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:53,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629862_1099518554492] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 12379 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 664729, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:53,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629845_1099518554475] hdfs.DFSClient: Slow ReadProcessor read fields took 137ms (threshold=20ms); ack: seqno: 38373 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 591650, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:45:53,728 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:45:53,766 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:53,789 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024715486 with entries=78072, filesize=122.38 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024753729 2015-10-16 12:45:53,792 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024700064 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024700064 2015-10-16 12:45:53,793 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966147, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:53,797 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024707592 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024707592 2015-10-16 12:45:54,124 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629863_1099518554493] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 13180 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870190, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:45:54,561 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629863_1099518554493] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 13756 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 793790, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:45:54,561 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 4090 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 520026, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:54,561 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:45:54,561 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:45:54,561 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:45:54,561 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:45:54,561 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:45:54,610 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 4113 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 691804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:55,105 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629863_1099518554493] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 14431 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 761656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:45:55,105 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 6875 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 457352, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:55,106 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:45:55,106 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:45:55,106 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:45:55,106 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:45:55,106 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:45:55,611 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 9706 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 456213, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:55,611 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629865_1099518554495] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 15063 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 785446, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:55,611 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:45:55,611 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:45:55,611 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:45:55,611 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:45:56,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629865_1099518554495] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 15712 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787884, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:56,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 12616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 461778, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:56,136 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:45:56,136 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:45:56,136 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:45:56,136 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:45:56,136 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:45:56,452 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 13933 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 519633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:56,692 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:45:56,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 15115 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 474235, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:56,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629865_1099518554495] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 16438 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 637725, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:45:56,692 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:45:56,692 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:45:56,692 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:45:56,692 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:45:57,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629866_1099518554496] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 17087 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 793041, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:57,212 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:57,212 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:45:57,212 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:45:57,212 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:57,213 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 18024 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 670268, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:57,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 20969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 490432, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:57,729 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629866_1099518554496] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 17719 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 770112, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:57,729 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:57,729 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:57,729 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:57,729 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:57,729 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:57,905 DEBUG [PriorityRpcServer.handler=9,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:45:57,905 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134270336 > lower bound=16777216 2015-10-16 12:45:57,906 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:45:57,933 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 22001 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 760986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:58,214 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629867_1099518554497] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 355 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:45:58,214 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 23278 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862591, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:58,215 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:58,215 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:58,215 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:58,215 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:45:58,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629866_1099518554496] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 18326 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023333, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:58,642 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:45:58,642 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:45:58,738 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3093249, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c9c29825a265427990f5b58289f97106 2015-10-16 12:45:58,744 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c9c29825a265427990f5b58289f97106 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c9c29825a265427990f5b58289f97106 2015-10-16 12:45:58,750 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c9c29825a265427990f5b58289f97106, entries=466300, sequenceid=3093249, filesize=70.8 M 2015-10-16 12:45:58,752 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134290080, currentsize=21.45 MB/22489520 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 847ms, sequenceid=3093249, compaction requested=false 2015-10-16 12:45:58,818 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 25906 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 55424646, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:58,818 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 56ms (threshold=20ms) 2015-10-16 12:45:58,818 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 56ms (threshold=20ms) 2015-10-16 12:45:58,818 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 56ms (threshold=20ms) 2015-10-16 12:45:58,818 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 56ms (threshold=20ms) 2015-10-16 12:45:58,818 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 56ms (threshold=20ms) 2015-10-16 12:45:59,136 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629868_1099518554498] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 19543 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 18131713, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:59,136 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 27673 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4568417, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:59,136 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:45:59,136 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:45:59,136 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:45:59,136 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:45:59,136 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:45:59,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629868_1099518554498] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 20220 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811068, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:45:59,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 30460 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 427708, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:45:59,678 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 64ms (threshold=20ms) 2015-10-16 12:45:59,678 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 64ms (threshold=20ms) 2015-10-16 12:45:59,678 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 64ms (threshold=20ms) 2015-10-16 12:45:59,678 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 64ms (threshold=20ms) 2015-10-16 12:45:59,678 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 64ms (threshold=20ms) 2015-10-16 12:45:59,821 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 30710 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 613007, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:46:00,193 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 32793 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 541566, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:46:00,193 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629869_1099518554499] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 20986 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 772511, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:00,193 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:00,193 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:00,193 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:00,193 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:00,726 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629869_1099518554499] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 21645 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 802138, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:00,726 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 35690 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 404527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:46:00,726 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:00,726 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:00,726 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:00,727 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:01,223 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629864_1099518554494] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 38484 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 593883, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:46:01,223 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629869_1099518554499] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 22310 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 707937, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:01,223 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:01,223 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:01,223 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:01,223 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:01,223 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:01,293 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:46:01,322 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024753729 with entries=77815, filesize=121.98 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024761293 2015-10-16 12:46:01,323 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024715486 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024715486 2015-10-16 12:46:01,732 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 2149 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 442703, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:01,732 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629871_1099518554501] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 23005 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 681640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:46:01,733 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:01,733 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:01,733 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:01,733 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:02,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629871_1099518554501] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 23703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 707087, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:46:02,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 4817 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 445383, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:02,248 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:02,248 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:02,248 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:02,248 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:02,248 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:02,529 DEBUG [PriorityRpcServer.handler=15,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:46:02,529 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134273312 > lower bound=16777216 2015-10-16 12:46:02,529 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:46:02,558 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 6391 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 625482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:02,755 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 7319 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 462896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:02,755 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629872_1099518554502] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861964, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:02,755 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629871_1099518554501] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 24334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 712137, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:46:02,756 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:02,756 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:02,756 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:02,756 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:02,756 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:02,968 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 7930 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 708147, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:03,236 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 9157 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 556516, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:03,236 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629871_1099518554501] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 24942 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 800869, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:46:03,236 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629872_1099518554502] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 805 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898207, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:03,236 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:03,237 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:03,237 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:03,236 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:03,237 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:03,488 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3139880, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/39664ff2549845bf9ff3cf21bfdef5da 2015-10-16 12:46:03,494 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/39664ff2549845bf9ff3cf21bfdef5da as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/39664ff2549845bf9ff3cf21bfdef5da 2015-10-16 12:46:03,564 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 10411 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 518676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:03,564 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 69ms (threshold=20ms) 2015-10-16 12:46:03,564 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 68ms (threshold=20ms) 2015-10-16 12:46:03,564 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 68ms (threshold=20ms) 2015-10-16 12:46:03,564 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 68ms (threshold=20ms) 2015-10-16 12:46:03,564 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629873_1099518554503] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 25283 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857551, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:46:03,569 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/39664ff2549845bf9ff3cf21bfdef5da, entries=466270, sequenceid=3139880, filesize=70.8 M 2015-10-16 12:46:03,571 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134282400, currentsize=22.42 MB/23514160 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1042ms, sequenceid=3139880, compaction requested=false 2015-10-16 12:46:03,785 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629873_1099518554503] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 25556 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 730596, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:46:03,785 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 11492 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 527608, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:03,785 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:03,785 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:03,785 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:03,785 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:03,785 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:04,315 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629873_1099518554503] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 26226 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801454, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:46:04,315 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 14294 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 462247, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:04,315 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:04,315 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:04,315 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:04,315 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:04,837 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629873_1099518554503] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 26879 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758867, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:46:04,837 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 17152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 722263, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:04,837 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:04,837 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:04,837 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:04,837 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:04,837 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:05,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629874_1099518554504] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 27538 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 743299, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:05,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 19939 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2196755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:05,363 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:05,363 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:05,363 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:05,363 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:05,363 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:05,899 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:05,899 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:05,899 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:05,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629874_1099518554504] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 28215 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 810570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:05,900 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:06,014 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 23066 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 590880, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:06,181 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 23498 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 686481, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:06,477 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629874_1099518554504] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 28997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 796809, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:06,477 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 24942 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 453456, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:06,477 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:06,477 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:06,477 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:06,477 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:06,477 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:06,997 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629875_1099518554505] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 29658 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 784957, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:46:06,997 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 27774 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 544522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:06,997 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:06,997 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:06,997 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:06,997 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:06,997 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:07,364 DEBUG [PriorityRpcServer.handler=12,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:46:07,365 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134273936 > lower bound=16777216 2015-10-16 12:46:07,365 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:46:07,394 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 29849 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 536113, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:07,516 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 30341 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 522397, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:07,516 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629876_1099518554506] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 121 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790485, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:46:07,516 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629875_1099518554505] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 30318 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 723140, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:46:07,516 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:07,516 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:07,516 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:07,516 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:07,516 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:07,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629876_1099518554506] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 740 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1119425, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:46:07,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629875_1099518554505] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 30862 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 850632, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:46:07,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 32475 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 538672, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:07,959 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:07,959 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:07,959 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:07,959 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:07,959 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:08,263 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3186512, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/36d6070bbb734c6ebbef8abd4ed5e034 2015-10-16 12:46:08,269 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/36d6070bbb734c6ebbef8abd4ed5e034 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/36d6070bbb734c6ebbef8abd4ed5e034 2015-10-16 12:46:08,291 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/36d6070bbb734c6ebbef8abd4ed5e034, entries=466280, sequenceid=3186512, filesize=70.8 M 2015-10-16 12:46:08,293 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134285040, currentsize=24.50 MB/25688960 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 928ms, sequenceid=3186512, compaction requested=true 2015-10-16 12:46:08,293 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:46:08,293 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 5 compacting, 3 eligible, 10 blocking 2015-10-16 12:46:08,295 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222834150 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 12:46:08,295 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:46:08,295 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:46:08,295 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 12:46:08,295 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c9c29825a265427990f5b58289f97106, keycount=466300, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=3093249 2015-10-16 12:46:08,296 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/39664ff2549845bf9ff3cf21bfdef5da, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=3139880 2015-10-16 12:46:08,296 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/36d6070bbb734c6ebbef8abd4ed5e034, keycount=466280, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=3186512 2015-10-16 12:46:08,311 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:46:08,393 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629877_1099518554507] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 31395 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 745800, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:08,393 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 34664 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 534323, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:08,393 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629878_1099518554508] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 54 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 760331, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:08,393 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:46:08,393 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:46:08,393 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:46:08,393 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:46:08,394 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:08,785 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629877_1099518554507] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 31817 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7116600, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:08,785 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629878_1099518554508] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 543 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2444088, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:08,785 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 36478 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2381109, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:08,786 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:08,786 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:08,786 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:08,786 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:08,786 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:09,180 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629878_1099518554508] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 989 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853692, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:09,180 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629870_1099518554500] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 38433 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 650461, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:09,180 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:09,180 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:09,181 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:09,181 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:09,181 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:09,181 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629877_1099518554507] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 32237 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 791303, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:09,281 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:46:09,310 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:46:09,330 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024761293 with entries=77903, filesize=122.12 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024769281 2015-10-16 12:46:09,332 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024753729 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024753729 2015-10-16 12:46:09,334 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 953407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:09,568 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629877_1099518554507] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 32713 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 715709, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:09,568 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629878_1099518554508] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1429 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 763797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:09,568 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1020 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 459403, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:09,568 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:09,568 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:09,568 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:09,568 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:09,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 2755 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 523394, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:09,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629877_1099518554507] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 33180 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804272, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:09,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629878_1099518554508] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1894 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 675623, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:09,932 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:46:09,932 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:46:09,932 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:46:09,932 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:46:09,932 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:46:10,054 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 2886 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 805858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:10,325 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629881_1099518554511] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 2425 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 980580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:10,326 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 3876 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736273, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:10,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 5466 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:10,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629881_1099518554511] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 2916 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974578, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:10,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629880_1099518554510] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 34177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1219118, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:10,692 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:10,692 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:10,692 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:10,692 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:10,692 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:11,065 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:11,065 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 7045 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 602638, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:11,065 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629881_1099518554511] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 3405 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127417, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:11,065 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629880_1099518554510] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 34649 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 976055, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:11,066 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:11,066 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:11,065 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:11,065 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:11,101 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#17 average throughput is 74.35 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 12:46:11,118 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c8faeced627342bf8fc52f5ee7e00d2d as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c8faeced627342bf8fc52f5ee7e00d2d 2015-10-16 12:46:11,125 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:46:11,128 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:46:11,137 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c9c29825a265427990f5b58289f97106, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c9c29825a265427990f5b58289f97106 2015-10-16 12:46:11,146 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/39664ff2549845bf9ff3cf21bfdef5da, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/39664ff2549845bf9ff3cf21bfdef5da 2015-10-16 12:46:11,154 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/36d6070bbb734c6ebbef8abd4ed5e034, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/36d6070bbb734c6ebbef8abd4ed5e034 2015-10-16 12:46:11,154 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into c8faeced627342bf8fc52f5ee7e00d2d(size=212.5 M), total size for store is 4.7 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-10-16 12:46:11,154 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=2, time=36902677148973507; duration=2sec 2015-10-16 12:46:11,154 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:46:11,598 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 9586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 430666, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:11,598 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629880_1099518554510] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 35335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:11,599 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:11,599 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:11,599 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:46:12,127 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 12399 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692762, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:12,127 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:12,127 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:12,127 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:12,127 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:12,127 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:12,426 DEBUG [PriorityRpcServer.handler=6,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:46:12,426 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134273136 > lower bound=16777216 2015-10-16 12:46:12,427 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:46:12,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 14011 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 684207, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:12,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629883_1099518554513] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 223 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833463, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:46:12,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 14875 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 471003, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:12,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629882_1099518554512] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 36630 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 754497, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:12,641 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:12,642 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:12,642 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:12,642 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:13,074 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629883_1099518554513] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 829 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:46:13,074 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629882_1099518554512] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 37162 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 780368, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:13,074 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 17037 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 430415, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:13,074 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:46:13,074 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:46:13,074 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:46:13,075 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:13,075 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:13,312 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3233145, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/70bfb9eb7e0d4825a21dba37e7516aae 2015-10-16 12:46:13,319 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/70bfb9eb7e0d4825a21dba37e7516aae as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/70bfb9eb7e0d4825a21dba37e7516aae 2015-10-16 12:46:13,341 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/70bfb9eb7e0d4825a21dba37e7516aae, entries=466280, sequenceid=3233145, filesize=70.8 M 2015-10-16 12:46:13,343 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134284240, currentsize=24.06 MB/25230800 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 917ms, sequenceid=3233145, compaction requested=false 2015-10-16 12:46:13,561 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:13,561 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:13,561 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:13,561 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629884_1099518554514] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 37772 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 717576, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:46:13,561 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:13,561 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:14,105 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629884_1099518554514] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 38473 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728328, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:46:14,105 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 22159 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 447746, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:14,105 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:14,105 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:14,105 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:14,105 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:14,105 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:14,650 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:14,650 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629884_1099518554514] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 39167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 679862, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:46:14,650 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 24907 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 571745, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:14,651 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:14,651 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:14,651 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:14,791 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 25172 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 630684, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:15,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629885_1099518554515] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 39931 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:15,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 27303 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 604688, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:15,208 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:15,209 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:15,208 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:15,208 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:15,208 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:15,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629885_1099518554515] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 40629 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767900, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:15,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 30026 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 590216, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:15,744 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:15,744 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:15,744 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:15,744 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:16,263 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:16,263 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:16,263 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 32600 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 706084, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:16,263 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629885_1099518554515] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 41340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 744160, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:16,263 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:16,263 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:16,263 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:16,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 34300 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 639405, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:16,644 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:16,644 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:16,644 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:16,644 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:16,644 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:16,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629886_1099518554516] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 42110 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 760040, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:46:16,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 34971 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1949904, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:16,796 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:16,796 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:16,796 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:16,796 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:16,796 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:17,310 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 37524 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 580398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:17,310 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629886_1099518554516] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 42833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 670157, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:46:17,310 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:17,310 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:17,310 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:17,310 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:17,310 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:17,311 DEBUG [PriorityRpcServer.handler=12,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:46:17,311 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134279216 > lower bound=16777216 2015-10-16 12:46:17,311 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:46:17,336 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629879_1099518554509] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 37537 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 633928, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:17,599 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:46:17,638 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024769281 with entries=77785, filesize=121.93 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024777599 2015-10-16 12:46:17,639 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024761293 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024761293 2015-10-16 12:46:17,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1123253, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:17,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 565 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 674189, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:17,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629886_1099518554516] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 43444 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 828303, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:46:17,766 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:17,766 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:17,766 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:17,766 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:17,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629887_1099518554517] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 427 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819377, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:17,767 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:18,529 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629887_1099518554517] hdfs.DFSClient: Slow ReadProcessor read fields took 288ms (threshold=20ms); ack: seqno: 1152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 287246068, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:18,529 WARN [MemStoreFlusher.1] hdfs.DFSClient: Slow waitForAckedSeqno took 289ms (threshold=20ms) 2015-10-16 12:46:18,537 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3279776, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/0e98c47963e04dc8a2e49fc9b64919b8 2015-10-16 12:46:18,543 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/0e98c47963e04dc8a2e49fc9b64919b8 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0e98c47963e04dc8a2e49fc9b64919b8 2015-10-16 12:46:18,549 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0e98c47963e04dc8a2e49fc9b64919b8, entries=466270, sequenceid=3279776, filesize=70.8 M 2015-10-16 12:46:18,550 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134281680, currentsize=33.52 MB/35152240 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1239ms, sequenceid=3279776, compaction requested=true 2015-10-16 12:46:18,550 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:46:18,551 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 5 compacting, 3 eligible, 10 blocking 2015-10-16 12:46:18,552 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:46:18,552 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:46:18,552 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:46:18,946 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 6368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 583213, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:18,947 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629889_1099518554519] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 44383 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758401, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:18,947 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:18,947 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:18,947 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:18,947 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:18,947 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:19,250 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 7414 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 593842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:19,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 8703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 587862, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:19,534 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:19,534 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:19,534 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:19,535 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:19,535 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:19,536 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629889_1099518554519] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 45164 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915243, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:20,086 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629889_1099518554519] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 45818 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787865, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:20,086 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 11545 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 639855, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:20,086 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:20,086 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:20,086 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:20,086 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:20,086 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:20,605 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 14174 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 695057, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:20,605 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629890_1099518554520] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 46518 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 925092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:20,606 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:20,606 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:20,606 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:20,606 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:20,606 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:21,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 16800 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 586851, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:21,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629890_1099518554520] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 47235 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 749027, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:21,128 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:21,128 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:21,128 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:21,128 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:21,128 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:21,656 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629891_1099518554521] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 47939 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4112576, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:21,656 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 19398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 693320, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:21,656 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:21,656 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:22,120 DEBUG [PriorityRpcServer.handler=10,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:46:22,121 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134272528 > lower bound=16777216 2015-10-16 12:46:22,121 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:46:22,148 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 21814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 552463, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:22,201 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 21937 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 510129, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:22,201 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629891_1099518554521] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 48664 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794441, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:22,201 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629892_1099518554522] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 14 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2608965, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:46:22,201 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:22,201 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:22,632 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629892_1099518554522] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 633 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:46:22,632 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 23978 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 570823, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:22,632 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:22,632 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:22,632 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:22,632 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:22,632 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629891_1099518554521] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 49232 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 720666, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:22,632 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:22,970 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3326409, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/4cb8375665ff4fb9ae0cb86a85d01ad9 2015-10-16 12:46:22,976 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/4cb8375665ff4fb9ae0cb86a85d01ad9 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4cb8375665ff4fb9ae0cb86a85d01ad9 2015-10-16 12:46:22,981 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4cb8375665ff4fb9ae0cb86a85d01ad9, entries=466290, sequenceid=3326409, filesize=70.8 M 2015-10-16 12:46:22,982 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134286800, currentsize=21.35 MB/22389040 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 861ms, sequenceid=3326409, compaction requested=true 2015-10-16 12:46:22,982 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:46:22,983 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 5 compacting, 4 eligible, 10 blocking 2015-10-16 12:46:22,984 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 445665413 starting at candidate #0 after considering 3 permutations with 2 in ratio 2015-10-16 12:46:22,984 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:46:22,984 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:46:22,985 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=425.0 M 2015-10-16 12:46:22,985 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c8faeced627342bf8fc52f5ee7e00d2d, keycount=1398850, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=3186512 2015-10-16 12:46:22,985 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/70bfb9eb7e0d4825a21dba37e7516aae, keycount=466280, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=3233145 2015-10-16 12:46:22,985 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0e98c47963e04dc8a2e49fc9b64919b8, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=3279776 2015-10-16 12:46:22,985 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4cb8375665ff4fb9ae0cb86a85d01ad9, keycount=466290, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=3326409 2015-10-16 12:46:23,004 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:46:23,086 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629893_1099518554523] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 64 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 823387, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:46:23,086 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629891_1099518554521] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 49764 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869826, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:23,086 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 25984 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 645061, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:23,086 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:23,086 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:23,086 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:23,086 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:23,086 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:23,503 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629894_1099518554524] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 50134 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945841, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:23,503 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629893_1099518554523] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 618 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 831088, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:46:23,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 27752 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842795, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:23,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629893_1099518554523] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 1140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 740805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:46:23,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629894_1099518554524] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 50532 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 851894, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:23,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 29454 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 635298, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:23,906 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:23,906 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:23,906 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:23,906 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:23,906 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:24,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629894_1099518554524] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 50723 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845575, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:24,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629893_1099518554523] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 1456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 733175, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:46:24,208 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 77ms (threshold=20ms) 2015-10-16 12:46:24,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 30623 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 497129, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:24,208 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 77ms (threshold=20ms) 2015-10-16 12:46:24,208 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 76ms (threshold=20ms) 2015-10-16 12:46:24,208 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 76ms (threshold=20ms) 2015-10-16 12:46:24,208 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 76ms (threshold=20ms) 2015-10-16 12:46:24,369 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 31328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 515761, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:24,369 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629893_1099518554523] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1644 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 687004, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:46:24,369 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629894_1099518554524] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 50910 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 785973, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:24,369 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:24,369 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:24,369 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:24,369 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:46:24,406 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629893_1099518554523] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 1646 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913079, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:46:24,747 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 33033 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 688448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:24,747 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629895_1099518554525] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 2096 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3274580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:46:24,747 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629894_1099518554524] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 51354 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 740283, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:24,747 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:24,748 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:25,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 34747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 522825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:25,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629894_1099518554524] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 51794 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 661505, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:25,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629895_1099518554525] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 2605 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 754713, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:46:25,120 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:25,120 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:25,120 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:25,120 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:25,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629895_1099518554525] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 3146 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 718314, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:46:25,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 36577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 564658, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:25,526 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:25,526 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:25,526 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:25,526 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:25,526 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:26,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629888_1099518554518] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 38750 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 588510, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:46:26,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629896_1099518554526] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 52339 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825365, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:46:26,003 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:26,003 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:26,003 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:26,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629895_1099518554525] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 3798 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932071, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:46:26,003 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:26,003 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:26,024 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:46:26,071 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024777599 with entries=77865, filesize=122.06 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024786024 2015-10-16 12:46:26,073 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024769281 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024769281 2015-10-16 12:46:26,074 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1143101, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:26,409 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:26,409 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:26,409 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629898_1099518554528] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 4328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692238, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:46:26,409 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:26,409 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:26,409 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:26,410 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629896_1099518554526] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 52730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089330, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:46:26,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 3266 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 602906, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:26,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629898_1099518554528] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 4811 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 700769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:46:26,783 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:26,783 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:26,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629896_1099518554526] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 53171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 648580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:46:26,783 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:46:26,783 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:26,783 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:27,163 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 4988 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 583911, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:27,163 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629896_1099518554526] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 53623 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 702184, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:46:27,163 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:27,163 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:27,163 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:27,163 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629898_1099518554528] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 5294 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 714438, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:46:27,163 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:27,163 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:27,400 DEBUG [PriorityRpcServer.handler=7,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:46:27,400 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134272272 > lower bound=16777216 2015-10-16 12:46:27,400 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:46:27,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 6142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792477, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:27,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629899_1099518554529] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 137 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 717498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:27,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629896_1099518554526] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 53989 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920921, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:46:27,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629898_1099518554528] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 5804 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841617, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:46:27,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 6568 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 621013, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:27,556 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:27,556 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:27,556 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:27,556 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:27,556 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:27,886 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629899_1099518554529] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 598 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896927, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:27,886 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629898_1099518554528] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 6211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1094661, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:46:27,886 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 7901 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 693877, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:27,886 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629900_1099518554530] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 54385 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:46:27,886 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:27,886 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:27,886 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:27,886 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:27,886 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:28,071 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 8268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 881163, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:28,242 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 8960 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 714635, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:28,242 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629901_1099518554531] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 6643 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865216, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:46:28,242 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:28,242 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:28,242 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:28,242 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629900_1099518554530] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 54815 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1031307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:46:28,242 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629899_1099518554529] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 1094 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841729, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:28,242 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:28,242 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:28,287 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3373041, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/719693e46e1f48eda540e95642f437b9 2015-10-16 12:46:28,295 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/719693e46e1f48eda540e95642f437b9 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/719693e46e1f48eda540e95642f437b9 2015-10-16 12:46:28,308 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/719693e46e1f48eda540e95642f437b9, entries=466280, sequenceid=3373041, filesize=70.8 M 2015-10-16 12:46:28,309 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134284240, currentsize=17.44 MB/18287760 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 909ms, sequenceid=3373041, compaction requested=false 2015-10-16 12:46:28,440 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#18 average throughput is 76.14 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 12:46:28,461 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5eec301127f84273841e8419ac0cecfa as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5eec301127f84273841e8419ac0cecfa 2015-10-16 12:46:28,475 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:46:28,479 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:46:28,486 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c8faeced627342bf8fc52f5ee7e00d2d, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c8faeced627342bf8fc52f5ee7e00d2d 2015-10-16 12:46:28,495 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/70bfb9eb7e0d4825a21dba37e7516aae, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/70bfb9eb7e0d4825a21dba37e7516aae 2015-10-16 12:46:28,503 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0e98c47963e04dc8a2e49fc9b64919b8, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0e98c47963e04dc8a2e49fc9b64919b8 2015-10-16 12:46:28,511 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4cb8375665ff4fb9ae0cb86a85d01ad9, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4cb8375665ff4fb9ae0cb86a85d01ad9 2015-10-16 12:46:28,511 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 5eec301127f84273841e8419ac0cecfa(size=425.0 M), total size for store is 5.0 G. This selection was in queue for 0sec, and took 5sec to execute. 2015-10-16 12:46:28,511 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=425.0 M, priority=2, time=36902691838438698; duration=5sec 2015-10-16 12:46:28,512 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:46:28,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 10951 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 475728, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:28,668 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629900_1099518554530] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 55366 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 716273, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:46:28,668 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:28,668 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:28,668 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:46:28,668 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:28,668 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:46:29,202 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:29,202 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:29,202 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629900_1099518554530] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 56092 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 716883, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:46:29,203 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 13549 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808125, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:29,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 16212 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 563726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:29,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629902_1099518554532] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 56804 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:29,733 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:29,733 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:29,733 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:29,733 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:29,733 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:30,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 18891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 482631, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:30,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629902_1099518554532] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 57495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 748840, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:30,258 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:30,258 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:30,258 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:30,258 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:30,258 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:30,820 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629902_1099518554532] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 58102 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741151, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:30,820 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 21887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 600989, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:30,820 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:30,820 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:30,820 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:30,820 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:30,820 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:31,528 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 487ms (threshold=20ms); ack: seqno: 23071 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 487739289, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:31,528 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 488ms (threshold=20ms) 2015-10-16 12:46:31,528 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 488ms (threshold=20ms) 2015-10-16 12:46:31,528 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 488ms (threshold=20ms) 2015-10-16 12:46:31,528 INFO [sync.3] wal.FSHLog: Slow sync cost: 487 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:31,528 INFO [sync.1] wal.FSHLog: Slow sync cost: 488 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:31,528 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 489ms (threshold=20ms) 2015-10-16 12:46:31,528 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 489ms (threshold=20ms) 2015-10-16 12:46:31,528 INFO [sync.4] wal.FSHLog: Slow sync cost: 488 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:31,528 INFO [sync.0] wal.FSHLog: Slow sync cost: 488 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:31,528 INFO [sync.2] wal.FSHLog: Slow sync cost: 488 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:31,528 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629903_1099518554533] hdfs.DFSClient: Slow ReadProcessor read fields took 488ms (threshold=20ms); ack: seqno: 58417 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 487414099, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:31,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 24417 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 487171, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:31,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629903_1099518554533] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 58832 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 743497, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:31,792 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:31,792 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:31,792 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:31,792 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:31,792 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:32,330 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629903_1099518554533] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 59579 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 695832, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:32,377 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 26952 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 623602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:32,815 DEBUG [PriorityRpcServer.handler=15,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:46:32,815 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134265696 > lower bound=16777216 2015-10-16 12:46:32,815 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:46:32,851 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629903_1099518554533] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 60312 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 663937, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:32,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 29464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 570261, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:33,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 31541 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 494536, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:33,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629904_1099518554534] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 574 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 739063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:46:33,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629905_1099518554535] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 60862 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 635574, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:46:33,283 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:33,283 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:33,283 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:33,283 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:33,283 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:33,718 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629905_1099518554535] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 61402 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 634527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:46:33,718 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 33762 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 496686, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:33,718 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:33,718 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:33,718 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:33,718 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:33,718 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:33,901 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629904_1099518554534] hdfs.DFSClient: Slow ReadProcessor read fields took 245ms (threshold=20ms); ack: seqno: 1152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 244367373, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:46:33,901 WARN [MemStoreFlusher.0] hdfs.DFSClient: Slow waitForAckedSeqno took 246ms (threshold=20ms) 2015-10-16 12:46:33,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3419669, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/88fab39a770a4f7e87b45c995ac17096 2015-10-16 12:46:33,917 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/88fab39a770a4f7e87b45c995ac17096 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/88fab39a770a4f7e87b45c995ac17096 2015-10-16 12:46:33,923 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/88fab39a770a4f7e87b45c995ac17096, entries=466230, sequenceid=3419669, filesize=70.8 M 2015-10-16 12:46:33,925 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134271040, currentsize=30.46 MB/31938000 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1110ms, sequenceid=3419669, compaction requested=true 2015-10-16 12:46:33,925 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:46:33,925 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 5 compacting, 3 eligible, 10 blocking 2015-10-16 12:46:33,926 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:46:33,926 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:46:33,927 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:46:34,212 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:34,212 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:34,212 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:34,212 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:34,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629905_1099518554535] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 62041 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3660400, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:46:34,212 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:34,475 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 37531 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 595919, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:34,475 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 105ms (threshold=20ms) 2015-10-16 12:46:34,475 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 105ms (threshold=20ms) 2015-10-16 12:46:34,475 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 105ms (threshold=20ms) 2015-10-16 12:46:34,475 INFO [sync.0] wal.FSHLog: Slow sync cost: 105 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:34,475 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 105ms (threshold=20ms) 2015-10-16 12:46:34,475 INFO [sync.4] wal.FSHLog: Slow sync cost: 104 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:34,475 INFO [sync.2] wal.FSHLog: Slow sync cost: 104 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:34,475 INFO [sync.3] wal.FSHLog: Slow sync cost: 104 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:34,475 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 105ms (threshold=20ms) 2015-10-16 12:46:34,475 INFO [sync.1] wal.FSHLog: Slow sync cost: 104 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:34,701 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:46:34,743 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:34,774 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629906_1099518554536] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 62793 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1476044, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:46:34,774 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629897_1099518554527] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 39020 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 998357, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:34,774 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:34,774 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:34,774 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:34,774 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:34,774 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:34,787 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024786024 with entries=77993, filesize=122.26 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024794702 2015-10-16 12:46:34,788 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024777599 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024777599 2015-10-16 12:46:34,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967812, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:35,315 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629906_1099518554536] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 63513 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 785618, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:46:35,315 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 2627 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:35,315 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:35,315 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:35,315 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:35,315 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:35,315 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:35,837 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 5347 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 714452, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:35,837 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629906_1099518554536] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 64199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 771222, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:46:35,837 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:35,838 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:35,838 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:35,837 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:35,837 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:36,052 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 5944 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 777107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:36,403 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 7596 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 463284, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:36,404 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629908_1099518554538] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 64986 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 694319, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:46:36,404 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:36,404 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:36,404 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:36,404 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:36,404 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:36,917 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 10350 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 436924, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:36,918 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:36,918 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:36,918 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:36,918 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629908_1099518554538] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 65664 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 688647, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:46:37,437 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 13129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 594515, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:37,437 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629908_1099518554538] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 66342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 757847, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:46:37,437 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:37,437 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:37,437 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:37,437 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:37,437 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:37,553 DEBUG [PriorityRpcServer.handler=2,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:46:37,554 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134264416 > lower bound=16777216 2015-10-16 12:46:37,554 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:46:37,580 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 13826 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 629552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:37,895 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:37,895 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629909_1099518554539] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 427 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834389, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:37,895 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629910_1099518554540] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 66944 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 970219, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:37,895 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 15275 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 453971, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:37,896 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:37,896 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:37,896 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:37,896 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:38,330 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629909_1099518554539] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 1065 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830641, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:38,330 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 17371 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 548936, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:38,331 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629910_1099518554540] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 67494 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 901922, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:38,331 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:38,331 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:38,331 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:38,331 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:38,331 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:38,394 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3466295, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ee31aa2fe3614641be8a13648b0256f9 2015-10-16 12:46:38,400 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ee31aa2fe3614641be8a13648b0256f9 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ee31aa2fe3614641be8a13648b0256f9 2015-10-16 12:46:38,405 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ee31aa2fe3614641be8a13648b0256f9, entries=466220, sequenceid=3466295, filesize=70.8 M 2015-10-16 12:46:38,407 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134266880, currentsize=21.78 MB/22837760 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 853ms, sequenceid=3466295, compaction requested=true 2015-10-16 12:46:38,407 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:46:38,407 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 5 compacting, 4 eligible, 10 blocking 2015-10-16 12:46:38,409 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222815057 starting at candidate #1 after considering 3 permutations with 1 in ratio 2015-10-16 12:46:38,409 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:46:38,409 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:46:38,409 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 12:46:38,409 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/719693e46e1f48eda540e95642f437b9, keycount=466280, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=3373041 2015-10-16 12:46:38,409 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/88fab39a770a4f7e87b45c995ac17096, keycount=466230, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=3419669 2015-10-16 12:46:38,409 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ee31aa2fe3614641be8a13648b0256f9, keycount=466220, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=3466295 2015-10-16 12:46:38,423 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:46:38,848 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629910_1099518554540] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 68096 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 748187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:38,848 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 19901 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 524302, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:38,848 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.8571428571428571, tune compaction throughput to 18.57 MB/sec 2015-10-16 12:46:38,848 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:38,848 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:38,848 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:38,848 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:38,848 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:39,506 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629911_1099518554541] hdfs.DFSClient: Slow ReadProcessor read fields took 256ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966971, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:39,646 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 24293 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 596455, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:39,646 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629911_1099518554541] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 309 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:39,646 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:39,646 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:39,646 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:39,646 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:39,646 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:39,797 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629910_1099518554540] hdfs.DFSClient: Slow ReadProcessor read fields took 921ms (threshold=20ms); ack: seqno: 68131 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3989658, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:40,361 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 27556 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 700557, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:40,582 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629911_1099518554541] hdfs.DFSClient: Slow ReadProcessor read fields took 920ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1153713, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:40,638 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 28912 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 580973, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:40,638 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629911_1099518554541] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861520, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:40,638 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:40,638 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:40,638 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:40,638 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:40,638 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:40,871 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629910_1099518554540] hdfs.DFSClient: Slow ReadProcessor read fields took 953ms (threshold=20ms); ack: seqno: 68298 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1185245, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:41,469 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 33305 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 414694, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:41,469 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:41,469 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:41,469 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:41,469 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:41,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629911_1099518554541] hdfs.DFSClient: Slow ReadProcessor read fields took 929ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1120884, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:41,948 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629910_1099518554540] hdfs.DFSClient: Slow ReadProcessor read fields took 958ms (threshold=20ms); ack: seqno: 68465 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1131171, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:42,217 DEBUG [PriorityRpcServer.handler=8,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:46:42,217 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134273456 > lower bound=16777216 2015-10-16 12:46:42,217 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:46:42,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 37253 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 576496, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:42,316 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629907_1099518554537] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 37410 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 678758, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:42,316 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:42,316 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:42,316 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:42,316 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:42,316 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629912_1099518554542] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 19 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1388153, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:42,316 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:42,604 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:46:42,636 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024794702 with entries=77820, filesize=121.99 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024802604 2015-10-16 12:46:42,639 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024786024 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024786024 2015-10-16 12:46:42,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629911_1099518554541] hdfs.DFSClient: Slow ReadProcessor read fields took 962ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1814824, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:42,936 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629913_1099518554543] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 1465 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 589766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:42,936 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629912_1099518554542] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 965 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 714328, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:42,936 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:42,936 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:42,936 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:46:42,936 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:42,936 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:46:43,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629910_1099518554540] hdfs.DFSClient: Slow ReadProcessor read fields took 959ms (threshold=20ms); ack: seqno: 68633 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1000598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:43,350 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629910_1099518554540] hdfs.DFSClient: Slow ReadProcessor read fields took 233ms (threshold=20ms); ack: seqno: 68770 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 232687258, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:43,351 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629912_1099518554542] hdfs.DFSClient: Slow ReadProcessor read fields took 286ms (threshold=20ms); ack: seqno: 1152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 285655667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:46:43,351 WARN [MemStoreFlusher.0] hdfs.DFSClient: Slow waitForAckedSeqno took 287ms (threshold=20ms) 2015-10-16 12:46:43,360 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3512931, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/27f9fdba80de429dab8ea75581c3b170 2015-10-16 12:46:43,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/27f9fdba80de429dab8ea75581c3b170 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/27f9fdba80de429dab8ea75581c3b170 2015-10-16 12:46:43,381 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/27f9fdba80de429dab8ea75581c3b170, entries=466320, sequenceid=3512931, filesize=70.8 M 2015-10-16 12:46:43,382 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134296080, currentsize=31.85 MB/33398480 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1165ms, sequenceid=3512931, compaction requested=false 2015-10-16 12:46:43,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 201ms (threshold=20ms); ack: seqno: 68796 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 203237070, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:43,772 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629913_1099518554543] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 5892 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 459722, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:43,773 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:43,773 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:43,773 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:43,773 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:43,811 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629911_1099518554541] hdfs.DFSClient: Slow ReadProcessor read fields took 950ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1515810, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:44,100 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 511ms (threshold=20ms); ack: seqno: 68803 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1479229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:44,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629913_1099518554543] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 10108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 457252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:44,623 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:44,623 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:44,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629911_1099518554541] hdfs.DFSClient: Slow ReadProcessor read fields took 948ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1361942, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:45,124 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629913_1099518554543] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 12428 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 569971, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:45,124 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 77ms (threshold=20ms) 2015-10-16 12:46:45,124 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 77ms (threshold=20ms) 2015-10-16 12:46:45,125 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 77ms (threshold=20ms) 2015-10-16 12:46:45,125 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 77ms (threshold=20ms) 2015-10-16 12:46:45,175 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 957ms (threshold=20ms); ack: seqno: 68970 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1055170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:45,198 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#16 1043706/1398730 (74.62%), rate=70348.70 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=18.57 MB/sec, activeCompactions=2] 2015-10-16 12:46:45,315 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#16 sleep 935 ms because current throughput is 70.92 MB/sec, max allowed is 9.29 MB/sec, already slept 6 time(s) and total slept time is 5699 ms till now. 2015-10-16 12:46:45,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629913_1099518554543] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 14335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 553430, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:45,519 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:45,519 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:45,519 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:45,519 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:45,519 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:45,683 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629913_1099518554543] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 14538 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:45,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629911_1099518554541] hdfs.DFSClient: Slow ReadProcessor read fields took 957ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1376172, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:46,252 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 936ms (threshold=20ms); ack: seqno: 69137 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1177498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:46,451 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629913_1099518554543] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 18459 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 603951, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:46,451 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:46,451 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:46,983 DEBUG [PriorityRpcServer.handler=16,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:46:46,983 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134279136 > lower bound=16777216 2015-10-16 12:46:46,984 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:46:47,010 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629913_1099518554543] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 21450 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 685432, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:47,040 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629911_1099518554541] hdfs.DFSClient: Slow ReadProcessor read fields took 959ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1332789, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:47,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629915_1099518554545] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 326 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835559, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:46:47,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629913_1099518554543] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 22721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 580763, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:47,305 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:47,306 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:47,306 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:46:47,306 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:47,306 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:47,329 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 937ms (threshold=20ms); ack: seqno: 69304 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1355433, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:47,819 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3559563, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8d5cfa97ec3d465da1aa1060993f2b31 2015-10-16 12:46:47,825 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8d5cfa97ec3d465da1aa1060993f2b31 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8d5cfa97ec3d465da1aa1060993f2b31 2015-10-16 12:46:47,831 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8d5cfa97ec3d465da1aa1060993f2b31, entries=466280, sequenceid=3559563, filesize=70.8 M 2015-10-16 12:46:47,832 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134284480, currentsize=21.27 MB/22307520 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 848ms, sequenceid=3559563, compaction requested=true 2015-10-16 12:46:47,832 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:46:47,977 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629913_1099518554543] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 25982 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 533604, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:47,977 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:47,977 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:47,977 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:47,977 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:47,977 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:48,117 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629911_1099518554541] hdfs.DFSClient: Slow ReadProcessor read fields took 916ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1373236, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:48,404 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 927ms (threshold=20ms); ack: seqno: 69471 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:48,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629913_1099518554543] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 30194 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 598875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:48,800 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:48,800 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:48,800 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:48,800 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:46:48,800 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:46:49,192 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629911_1099518554541] hdfs.DFSClient: Slow ReadProcessor read fields took 957ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1195022, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:49,480 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 932ms (threshold=20ms); ack: seqno: 69638 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1205159, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:49,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 69761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 770221, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:49,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629913_1099518554543] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 34531 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 525238, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:46:49,623 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:49,623 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:49,623 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:49,623 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:49,623 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:50,268 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629911_1099518554541] hdfs.DFSClient: Slow ReadProcessor read fields took 945ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1147667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:50,432 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:46:50,469 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024802604 with entries=77863, filesize=122.06 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024810432 2015-10-16 12:46:50,470 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024794702 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024794702 2015-10-16 12:46:50,562 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629916_1099518554546] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 322 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 630188, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:50,562 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:50,562 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:50,562 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:46:50,564 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 903ms (threshold=20ms); ack: seqno: 69805 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1245464, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:50,633 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 69806 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2143136, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:50,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629916_1099518554546] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 1992 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 784297, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:51,344 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629911_1099518554541] hdfs.DFSClient: Slow ReadProcessor read fields took 965ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1077763, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:51,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629916_1099518554546] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 4865 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 539207, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:51,455 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:51,455 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:51,455 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:51,455 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:51,640 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 910ms (threshold=20ms); ack: seqno: 69972 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1281385, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:51,646 DEBUG [PriorityRpcServer.handler=14,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:46:51,646 WARN [MemStoreFlusher.0] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:46:51,647 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 12:46:51,965 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629916_1099518554546] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 7057 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 799094, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:52,421 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629917_1099518554547] hdfs.DFSClient: Slow ReadProcessor read fields took 411ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2236934, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:52,456 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:52,456 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:52,456 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:52,456 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629916_1099518554546] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 9487 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 539775, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:52,456 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:52,456 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:52,456 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629917_1099518554547] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 2178 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1123599, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:52,716 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 934ms (threshold=20ms); ack: seqno: 70139 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1626264, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:53,293 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629916_1099518554546] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 13776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 703283, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:53,293 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:53,293 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:53,293 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:53,293 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:53,293 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:46:53,496 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629917_1099518554547] hdfs.DFSClient: Slow ReadProcessor read fields took 928ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1372298, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:53,793 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 947ms (threshold=20ms); ack: seqno: 70308 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1456477, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:54,151 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629916_1099518554546] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 17997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489164, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:54,151 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:54,152 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:54,572 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629917_1099518554547] hdfs.DFSClient: Slow ReadProcessor read fields took 964ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1231107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:54,868 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 950ms (threshold=20ms); ack: seqno: 70475 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1142256, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:55,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629916_1099518554546] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 22147 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 516423, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:55,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 70628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 732325, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:55,018 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:55,018 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:55,018 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:55,945 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 919ms (threshold=20ms); ack: seqno: 70641 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1399393, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:56,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629916_1099518554546] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 27090 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 576383, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:56,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 70710 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728009, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:56,031 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:46:56,031 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:46:56,031 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:46:56,031 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:46:56,031 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:46:56,267 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629917_1099518554547] hdfs.DFSClient: Slow ReadProcessor read fields took 1575ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 620106545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:56,724 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629917_1099518554547] hdfs.DFSClient: Slow ReadProcessor read fields took 395ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1247997, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:56,867 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629916_1099518554546] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 31204 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 644954, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:56,867 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629917_1099518554547] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 3007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 771606, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:56,867 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:56,867 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:56,867 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:56,867 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:56,867 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:46:57,020 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629914_1099518554544] hdfs.DFSClient: Slow ReadProcessor read fields took 917ms (threshold=20ms); ack: seqno: 70807 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1280333, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:57,829 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629916_1099518554546] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 36079 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 506072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:57,829 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:57,829 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:57,829 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:57,829 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:57,829 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:57,832 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629917_1099518554547] hdfs.DFSClient: Slow ReadProcessor read fields took 962ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1969616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:58,096 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 955ms (threshold=20ms); ack: seqno: 70980 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1160405, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:46:58,161 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629916_1099518554546] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 37239 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 821904, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:46:58,516 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:46:58,560 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024810432 with entries=77881, filesize=122.09 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024818516 2015-10-16 12:46:58,563 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629919_1099518554549] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1071278, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:58,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629919_1099518554549] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 980 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 411557, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:58,781 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:46:58,781 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:46:58,781 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:46:58,781 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:46:58,907 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629917_1099518554547] hdfs.DFSClient: Slow ReadProcessor read fields took 966ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1151556, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:59,013 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629917_1099518554547] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 3178 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2097611, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:46:59,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 957ms (threshold=20ms); ack: seqno: 71147 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1120700, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:46:59,558 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629919_1099518554549] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 5195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 486495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:46:59,558 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:59,558 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:59,558 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:46:59,558 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:59,558 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:46:59,983 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629917_1099518554547] hdfs.DFSClient: Slow ReadProcessor read fields took 909ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069241, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:00,061 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#19 average throughput is 9.55 MB/sec, slept 20 time(s) and total slept time is 17163 ms. 1 active compactions remaining, total limit is 18.57 MB/sec 2015-10-16 12:47:00,081 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/580a0b059e6044c282eed9675e296b48 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/580a0b059e6044c282eed9675e296b48 2015-10-16 12:47:00,089 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:47:00,094 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:47:00,101 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/719693e46e1f48eda540e95642f437b9, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/719693e46e1f48eda540e95642f437b9 2015-10-16 12:47:00,109 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/88fab39a770a4f7e87b45c995ac17096, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/88fab39a770a4f7e87b45c995ac17096 2015-10-16 12:47:00,117 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ee31aa2fe3614641be8a13648b0256f9, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ee31aa2fe3614641be8a13648b0256f9 2015-10-16 12:47:00,118 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 580a0b059e6044c282eed9675e296b48(size=212.5 M), total size for store is 5.3 G. This selection was in queue for 0sec, and took 21sec to execute. 2015-10-16 12:47:00,118 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=2, time=36902707262868947; duration=21sec 2015-10-16 12:47:00,118 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 12:47:00,118 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 5 compacting, 4 eligible, 10 blocking 2015-10-16 12:47:00,120 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio 2015-10-16 12:47:00,120 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:47:00,120 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:47:00,120 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 5 compacting, 4 eligible, 10 blocking 2015-10-16 12:47:00,122 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio 2015-10-16 12:47:00,122 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:47:00,122 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:47:00,250 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 953ms (threshold=20ms); ack: seqno: 71314 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050317, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:47:00,400 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 71471 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 662868, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:47:00,400 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629919_1099518554549] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 9615 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 618130, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:00,400 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:00,400 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:00,400 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:00,400 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:00,648 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=387527376 > lower bound=16777216 2015-10-16 12:47:00,649 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=369.52 MB 2015-10-16 12:47:00,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629919_1099518554549] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 11031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 786207, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:00,788 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 382ms (threshold=20ms); ack: seqno: 71481 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050964, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:47:01,197 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629920_1099518554550] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 673 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 688124, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:47:01,197 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629919_1099518554549] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 13179 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 401586, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:01,197 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:01,197 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:01,198 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:01,198 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:01,198 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:01,326 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 417ms (threshold=20ms); ack: seqno: 71648 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1001284, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:47:01,853 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629920_1099518554550] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 1651 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692669, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:47:01,853 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629919_1099518554549] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 16245 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 867205, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:01,853 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:47:01,853 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:47:01,853 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:47:01,853 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:47:01,853 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:47:01,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 415ms (threshold=20ms); ack: seqno: 71814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1099156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:47:02,102 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629920_1099518554550] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 1950 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 56695077, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:47:02,402 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 420ms (threshold=20ms); ack: seqno: 71981 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:47:02,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629921_1099518554551] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 2557 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 768981, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:02,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629919_1099518554549] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 19158 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 739677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:02,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 72022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 696156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:47:02,461 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:47:02,461 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:47:02,461 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:47:02,461 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:47:02,461 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:47:02,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 383ms (threshold=20ms); ack: seqno: 72148 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1168204, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:47:02,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3694131, memsize=369.6 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/2620664d2de24dfb94d746171e29f6fa 2015-10-16 12:47:02,973 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/2620664d2de24dfb94d746171e29f6fa as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2620664d2de24dfb94d746171e29f6fa 2015-10-16 12:47:02,979 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2620664d2de24dfb94d746171e29f6fa, entries=1345630, sequenceid=3694131, filesize=204.4 M 2015-10-16 12:47:02,981 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~369.58 MB/387529840, currentsize=59.82 MB/62727200 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2333ms, sequenceid=3694131, compaction requested=true 2015-10-16 12:47:02,981 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:47:02,981 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 5 compacting, 5 eligible, 10 blocking 2015-10-16 12:47:02,983 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 5 files of size 1031369623 starting at candidate #0 after considering 6 permutations with 2 in ratio 2015-10-16 12:47:02,983 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:47:02,983 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:47:02,984 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=983.6 M 2015-10-16 12:47:02,984 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5eec301127f84273841e8419ac0cecfa, keycount=2797690, bloomtype=ROW, size=425.0 M, encoding=NONE, seqNum=3326409 2015-10-16 12:47:02,984 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/580a0b059e6044c282eed9675e296b48, keycount=1398730, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=3466295 2015-10-16 12:47:02,984 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/27f9fdba80de429dab8ea75581c3b170, keycount=466320, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=3512931 2015-10-16 12:47:02,984 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8d5cfa97ec3d465da1aa1060993f2b31, keycount=466280, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=3559563 2015-10-16 12:47:02,984 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2620664d2de24dfb94d746171e29f6fa, keycount=1345630, bloomtype=ROW, size=204.4 M, encoding=NONE, seqNum=3694131 2015-10-16 12:47:03,007 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:47:03,064 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629919_1099518554549] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 22093 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 551045, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:03,064 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 72267 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1126340, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:47:03,064 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 6 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4173953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:03,064 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:47:03,064 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:47:03,065 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:47:03,065 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:47:03,065 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:47:03,983 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629919_1099518554549] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 26598 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907323, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:04,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 901ms (threshold=20ms); ack: seqno: 72315 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1151534, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:47:04,057 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629919_1099518554549] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 26829 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 501890, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:04,057 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 72326 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017886, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:47:04,057 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:04,057 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:04,057 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:04,057 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:04,057 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:04,096 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 923ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1370866, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:04,874 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629919_1099518554549] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 31169 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 527131, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:04,874 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:04,874 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:04,875 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:04,875 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:04,874 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:05,093 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 916ms (threshold=20ms); ack: seqno: 72482 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021746, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:47:05,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 946ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072680, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:05,463 DEBUG [PriorityRpcServer.handler=9,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:47:05,463 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134278496 > lower bound=16777216 2015-10-16 12:47:05,463 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:47:05,492 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629919_1099518554549] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 34382 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 603755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:05,647 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:05,647 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:05,647 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:05,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629923_1099518554553] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1494269, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:47:05,648 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629919_1099518554549] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 34999 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801366, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:06,169 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 950ms (threshold=20ms); ack: seqno: 72649 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1076569, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:47:06,247 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 949ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:06,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1063785, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:06,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 72777 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:47:06,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629919_1099518554549] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 38089 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 649042, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:06,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629923_1099518554553] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 1143 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 829441, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:47:06,291 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:06,291 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:06,291 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:06,292 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:47:06,292 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:06,339 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629923_1099518554553] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 1152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1191416, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:47:06,339 WARN [MemStoreFlusher.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:47:06,342 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3740762, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8d18242e35ad47e989925dbf6f3c37ec 2015-10-16 12:47:06,348 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8d18242e35ad47e989925dbf6f3c37ec as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8d18242e35ad47e989925dbf6f3c37ec 2015-10-16 12:47:06,354 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8d18242e35ad47e989925dbf6f3c37ec, entries=466270, sequenceid=3740762, filesize=70.8 M 2015-10-16 12:47:06,357 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134280960, currentsize=22.45 MB/23540320 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 894ms, sequenceid=3740762, compaction requested=false 2015-10-16 12:47:06,440 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:47:06,484 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024818516 with entries=77996, filesize=122.26 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024826440 2015-10-16 12:47:06,486 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024802604 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024802604 2015-10-16 12:47:06,487 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629924_1099518554554] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1159911, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:06,493 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024810432 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024810432 2015-10-16 12:47:07,176 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629924_1099518554554] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 3623 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 579037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:07,176 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:07,176 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:07,176 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:07,176 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:07,176 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:07,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629918_1099518554548] hdfs.DFSClient: Slow ReadProcessor read fields took 924ms (threshold=20ms); ack: seqno: 72815 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1091542, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:47:07,324 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 912ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1221694, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:07,985 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629924_1099518554554] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 7850 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 475727, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:07,985 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:07,985 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:07,985 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:07,985 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:07,985 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:08,321 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629925_1099518554555] hdfs.DFSClient: Slow ReadProcessor read fields took 913ms (threshold=20ms); ack: seqno: 72988 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1557336, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:08,400 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 957ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127050, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:08,752 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629924_1099518554554] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 12073 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 576430, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:08,752 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:08,753 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:08,753 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:08,753 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:09,359 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629924_1099518554554] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 14895 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985358, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:09,397 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629925_1099518554555] hdfs.DFSClient: Slow ReadProcessor read fields took 948ms (threshold=20ms); ack: seqno: 73156 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1398514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:09,476 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 957ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1223592, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:09,645 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629924_1099518554554] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 16219 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 630351, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:09,646 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:09,646 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:09,646 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:09,646 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:09,646 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:10,118 DEBUG [PriorityRpcServer.handler=0,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:47:10,118 WARN [MemStoreFlusher.0] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:47:10,118 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:47:10,473 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629925_1099518554555] hdfs.DFSClient: Slow ReadProcessor read fields took 952ms (threshold=20ms); ack: seqno: 73323 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1093189, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:10,552 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 954ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1077893, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:10,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629925_1099518554555] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 73418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 854637, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:10,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629924_1099518554554] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 21133 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 542595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:10,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 1210 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728007, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:10,623 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:10,623 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:10,623 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:10,623 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:10,623 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:11,572 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:11,572 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:47:11,572 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:47:11,572 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:11,572 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629924_1099518554554] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 25906 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2170309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:11,572 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:11,574 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629925_1099518554555] hdfs.DFSClient: Slow ReadProcessor read fields took 850ms (threshold=20ms); ack: seqno: 73490 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1442498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:11,629 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 915ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1345871, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:12,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629924_1099518554554] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 30108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 574591, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:12,413 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:12,413 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:12,413 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:12,413 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:12,413 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:12,650 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629925_1099518554555] hdfs.DFSClient: Slow ReadProcessor read fields took 946ms (threshold=20ms); ack: seqno: 73657 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1264641, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:12,704 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 945ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1086294, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:13,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629924_1099518554554] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 34352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 478107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:13,282 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:13,282 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:13,282 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:13,282 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:13,726 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629925_1099518554555] hdfs.DFSClient: Slow ReadProcessor read fields took 962ms (threshold=20ms); ack: seqno: 73824 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1173888, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:13,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 944ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 977588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:14,118 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629924_1099518554554] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 38561 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 526832, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:14,118 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:14,118 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:14,118 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:14,119 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:14,119 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:14,190 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:47:14,242 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024826440 with entries=77775, filesize=121.92 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024834190 2015-10-16 12:47:14,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629926_1099518554556] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1108405, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:14,724 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629926_1099518554556] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 2119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 691366, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:14,802 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629925_1099518554555] hdfs.DFSClient: Slow ReadProcessor read fields took 951ms (threshold=20ms); ack: seqno: 73991 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1224171, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:14,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 949ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1015646, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:15,069 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629926_1099518554556] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 3718 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 619947, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:15,069 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:15,069 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:15,069 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:47:15,069 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:47:15,069 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:47:15,878 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629925_1099518554555] hdfs.DFSClient: Slow ReadProcessor read fields took 950ms (threshold=20ms); ack: seqno: 74158 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054339, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:15,934 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 949ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1187709, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:16,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629926_1099518554556] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 8638 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 684483, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:16,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629925_1099518554555] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 74272 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 815158, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:16,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629922_1099518554552] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 2039 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 747395, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:16,003 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:16,003 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:16,003 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:16,003 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:16,003 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:47:16,925 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629926_1099518554556] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 13505 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 618690, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:16,925 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:47:16,925 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:47:16,925 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:47:16,925 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:47:16,925 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:47:16,954 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629925_1099518554555] hdfs.DFSClient: Slow ReadProcessor read fields took 912ms (threshold=20ms); ack: seqno: 74325 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1195054, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:17,009 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 884ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1272694, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:17,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629926_1099518554556] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 17721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 480373, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:17,766 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:47:17,766 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:47:17,766 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:47:18,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629925_1099518554555] hdfs.DFSClient: Slow ReadProcessor read fields took 930ms (threshold=20ms); ack: seqno: 74492 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1230252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:18,085 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 924ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1424398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:18,159 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 2354 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1032667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:18,584 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629926_1099518554556] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 21927 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 551569, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:18,584 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:18,584 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:18,585 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:18,585 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:18,585 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:19,107 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629925_1099518554555] hdfs.DFSClient: Slow ReadProcessor read fields took 955ms (threshold=20ms); ack: seqno: 74659 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1178517, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:19,161 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1288700, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:19,412 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629926_1099518554556] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 26167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 437653, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:19,412 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:19,412 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:19,412 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:19,412 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:19,412 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:19,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629926_1099518554556] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 27340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 734533, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:20,182 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629925_1099518554555] hdfs.DFSClient: Slow ReadProcessor read fields took 953ms (threshold=20ms); ack: seqno: 74825 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1152368, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:20,237 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 942ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1061036, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:20,433 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:20,433 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:20,433 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:20,433 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:20,433 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:20,435 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 183ms (threshold=20ms); ack: seqno: 2688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1180319, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:20,494 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 2709 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947621, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:21,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629925_1099518554555] hdfs.DFSClient: Slow ReadProcessor read fields took 962ms (threshold=20ms); ack: seqno: 74992 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1138654, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:21,321 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629926_1099518554556] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 35884 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 498020, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:47:21,321 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:47:21,321 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:47:21,321 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:47:21,321 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:47:21,321 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:47:21,323 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 729ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1410196, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:21,933 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:47:21,983 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024834190 with entries=77950, filesize=122.20 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024841933 2015-10-16 12:47:21,987 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1211552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:22,151 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 753 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 404145, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:22,151 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:22,151 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:22,151 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:22,151 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:22,151 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:47:22,334 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629928_1099518554558] hdfs.DFSClient: Slow ReadProcessor read fields took 912ms (threshold=20ms); ack: seqno: 75162 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1260652, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:47:22,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 920ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1307053, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:22,450 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#16 average throughput is 46.27 MB/sec, slept 43 time(s) and total slept time is 37527 ms. 1 active compactions remaining, total limit is 18.57 MB/sec 2015-10-16 12:47:22,475 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c01239c636ca4b4eae6973f4079040fd as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c01239c636ca4b4eae6973f4079040fd 2015-10-16 12:47:22,495 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:47:22,499 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:47:22,508 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/53d2a7c0edf3492cadeec7d370728efb, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/53d2a7c0edf3492cadeec7d370728efb 2015-10-16 12:47:22,516 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1620b55306cf4c398136d5c37f71317d, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1620b55306cf4c398136d5c37f71317d 2015-10-16 12:47:22,524 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0afb955b872d42bbbf70f697b0ef7147, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0afb955b872d42bbbf70f697b0ef7147 2015-10-16 12:47:22,533 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d17768c1387840fc904553dd4e5b8395, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d17768c1387840fc904553dd4e5b8395 2015-10-16 12:47:22,541 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5766b7936c554fc5940d1b911ddb45c2, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5766b7936c554fc5940d1b911ddb45c2 2015-10-16 12:47:22,541 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 5 (all) file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into c01239c636ca4b4eae6973f4079040fd(size=4.5 G), total size for store is 5.5 G. This selection was in queue for 0sec, and took 1mins, 37sec to execute. 2015-10-16 12:47:22,542 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=5, fileSize=4.5 G, priority=5, time=36902654013441422; duration=1mins, 37sec 2015-10-16 12:47:22,542 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:47:22,542 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 5 compacting, 1 eligible, 10 blocking 2015-10-16 12:47:22,542 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:47:22,542 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:47:22,542 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:47:22,719 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=499351856 > lower bound=16777216 2015-10-16 12:47:22,719 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=476.17 MB 2015-10-16 12:47:22,822 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 4108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769369, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:22,961 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 4676 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 594689, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:22,961 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629930_1099518554560] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 118 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 791202, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:22,961 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:22,961 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:22,961 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:22,961 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:22,961 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:22,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 430ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1206363, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:23,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 411ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1181026, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:23,574 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:47:23,574 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:47:23,575 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629930_1099518554560] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 918 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736196, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:23,575 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 3403 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 721280, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:23,575 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 7761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 499836, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:23,575 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:47:23,575 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:47:23,575 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:47:24,038 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 376ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1142899, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:24,170 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629930_1099518554560] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 1784 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 822539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:24,170 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 3660 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845390, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:24,576 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 389ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1294790, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:25,115 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 401ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1433415, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:25,250 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 3892 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922856, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:25,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 3943 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1033358, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:25,388 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 3999 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:25,426 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629931_1099518554561] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 3889 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782157, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:25,653 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 258ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1339670, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:25,666 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3914163, memsize=476.2 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/0eb06cec35354048a5a8904f1e2fd289 2015-10-16 12:47:25,673 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/0eb06cec35354048a5a8904f1e2fd289 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0eb06cec35354048a5a8904f1e2fd289 2015-10-16 12:47:25,688 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0eb06cec35354048a5a8904f1e2fd289, entries=1733960, sequenceid=3914163, filesize=263.4 M 2015-10-16 12:47:25,690 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 1594ms (threshold=20ms); ack: seqno: 10601 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 840347, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:25,690 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~476.23 MB/499365840, currentsize=35.81 MB/37553840 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2971ms, sequenceid=3914163, compaction requested=true 2015-10-16 12:47:25,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:47:25,690 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 5 compacting, 2 eligible, 10 blocking 2015-10-16 12:47:25,691 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:47:25,692 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:47:25,692 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:47:25,763 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629927_1099518554557] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 4114 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 867626, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:25,847 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 4173 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1527088, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:26,190 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 340ms (threshold=20ms); ack: seqno: 4181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1677143, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:26,205 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 516ms (threshold=20ms); ack: seqno: 10602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 879610, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:26,326 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 4291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 933781, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:26,419 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 4346 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1028824, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:26,545 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 11999 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 665713, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:26,727 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 307ms (threshold=20ms); ack: seqno: 4348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1282907, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:26,847 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:47:26,847 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:47:26,847 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 13506 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 491423, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:26,847 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 4470 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 826315, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:26,847 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:47:26,847 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:47:27,266 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 384ms (threshold=20ms); ack: seqno: 4517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1273509, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:27,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 4676 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 25955081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:27,692 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:27,692 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:27,692 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:27,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 18197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:27,692 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:27,692 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:27,810 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 401ms (threshold=20ms); ack: seqno: 4684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 6977908, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:28,342 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 412ms (threshold=20ms); ack: seqno: 4851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1336763, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:28,504 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:28,504 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 22493 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 423171, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:28,504 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:28,504 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:28,504 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 4982 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767055, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:28,504 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:28,504 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:28,880 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 350ms (threshold=20ms); ack: seqno: 5018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1359301, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:29,374 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 27249 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 593058, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:29,374 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:47:29,374 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:47:29,374 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:47:29,374 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:47:29,418 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 415ms (threshold=20ms); ack: seqno: 5185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1272616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:29,428 DEBUG [PriorityRpcServer.handler=3,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:47:29,428 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134278688 > lower bound=16777216 2015-10-16 12:47:29,428 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:47:29,452 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 27535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 608405, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:29,956 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 406ms (threshold=20ms); ack: seqno: 5352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1208229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:30,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 30347 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 693758, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:30,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 5389 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 809698, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:30,017 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:30,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629934_1099518554564] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 816 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 645830, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:47:30,017 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:30,017 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:30,017 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:30,017 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:30,233 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=3960797, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/0c986601629c4788b349391937eb0ef5 2015-10-16 12:47:30,238 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/0c986601629c4788b349391937eb0ef5 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0c986601629c4788b349391937eb0ef5 2015-10-16 12:47:30,244 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0c986601629c4788b349391937eb0ef5, entries=466300, sequenceid=3960797, filesize=70.8 M 2015-10-16 12:47:30,246 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134290080, currentsize=21.41 MB/22452000 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 818ms, sequenceid=3960797, compaction requested=true 2015-10-16 12:47:30,246 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:47:30,246 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 5 compacting, 3 eligible, 10 blocking 2015-10-16 12:47:30,247 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:47:30,247 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:47:30,247 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:47:30,495 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 387ms (threshold=20ms); ack: seqno: 5521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1192755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:30,751 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 34023 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 481000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:30,752 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:30,752 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:31,029 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 34945 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 564264, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:31,032 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 415ms (threshold=20ms); ack: seqno: 5688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1293261, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:31,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 424ms (threshold=20ms); ack: seqno: 5855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1322074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:31,740 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629929_1099518554559] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 38596 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 552706, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:31,741 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:31,741 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:31,807 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:47:31,841 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024841933 with entries=77863, filesize=122.06 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024851808 2015-10-16 12:47:31,845 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024818516 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024818516 2015-10-16 12:47:31,849 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024826440 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024826440 2015-10-16 12:47:31,858 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024834190 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024834190 2015-10-16 12:47:32,138 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 568ms (threshold=20ms); ack: seqno: 5856 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 567854323, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:32,637 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:32,637 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629935_1099518554565] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 4030 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 564391, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:32,637 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:32,637 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:32,637 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:32,637 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:32,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629933_1099518554563] hdfs.DFSClient: Slow ReadProcessor read fields took 412ms (threshold=20ms); ack: seqno: 6189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1266209, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:33,277 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 415ms (threshold=20ms); ack: seqno: 6358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1566442, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:33,460 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629935_1099518554565] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 8174 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 658616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:33,460 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:33,460 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:33,460 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:33,460 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:33,460 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:33,815 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 409ms (threshold=20ms); ack: seqno: 6525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1482476, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:34,163 DEBUG [PriorityRpcServer.handler=0,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:47:34,163 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134279456 > lower bound=16777216 2015-10-16 12:47:34,163 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:47:34,190 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629935_1099518554565] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 11772 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 551179, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:34,383 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629937_1099518554567] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 210 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873832, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:47:34,383 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629935_1099518554565] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 12619 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 525484, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:34,383 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:34,383 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:34,384 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:34,385 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 449ms (threshold=20ms); ack: seqno: 6694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1371721, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:34,923 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 408ms (threshold=20ms); ack: seqno: 6861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1319958, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:34,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629937_1099518554567] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 659395, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:47:34,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 6914 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1065353, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:34,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629935_1099518554565] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 15706 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 522092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:34,995 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:34,995 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:34,995 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:47:34,995 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:34,995 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:35,091 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=4007427, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/86f01b04ebf64b2bb870402058dca36a 2015-10-16 12:47:35,096 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/86f01b04ebf64b2bb870402058dca36a as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/86f01b04ebf64b2bb870402058dca36a 2015-10-16 12:47:35,103 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/86f01b04ebf64b2bb870402058dca36a, entries=466260, sequenceid=4007427, filesize=70.8 M 2015-10-16 12:47:35,105 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134279040, currentsize=24.61 MB/25806640 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 942ms, sequenceid=4007427, compaction requested=true 2015-10-16 12:47:35,105 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:47:35,105 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 5 compacting, 4 eligible, 10 blocking 2015-10-16 12:47:35,107 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio 2015-10-16 12:47:35,107 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:47:35,107 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:47:35,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 370ms (threshold=20ms); ack: seqno: 7028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1602576, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:35,493 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629935_1099518554565] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 17728 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 581430, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:35,960 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629935_1099518554565] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 19925 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 685158, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:35,960 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:35,960 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:35,960 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:35,960 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:35,961 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:35,999 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 421ms (threshold=20ms); ack: seqno: 7195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1263629, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:36,096 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 7276 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:36,168 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 7333 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885696, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:36,537 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 351ms (threshold=20ms); ack: seqno: 7362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1236460, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:36,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 7417 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863457, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:36,775 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:47:36,775 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629935_1099518554565] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 24119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 646970, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:36,775 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:47:36,775 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:36,775 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:36,775 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:36,848 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:47:36,848 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:47:37,076 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 377ms (threshold=20ms); ack: seqno: 7529 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1345149, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:37,614 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 418ms (threshold=20ms); ack: seqno: 7696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1105358, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:37,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629935_1099518554565] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 28851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 604376, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:37,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 7746 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751750, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:37,685 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:37,685 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:37,685 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:37,685 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:37,685 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:47:38,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 383ms (threshold=20ms); ack: seqno: 7863 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1226164, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:38,564 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629935_1099518554565] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 33303 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 646017, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:38,564 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:38,564 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:38,564 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:38,564 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:38,564 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:38,690 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 418ms (threshold=20ms); ack: seqno: 8030 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1267363, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.0, tune compaction throughput to 20.00 MB/sec 2015-10-16 12:47:38,918 DEBUG [PriorityRpcServer.handler=19,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:47:38,918 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134282608 > lower bound=16777216 2015-10-16 12:47:38,918 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:47:38,944 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629935_1099518554565] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 35186 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 516837, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:39,229 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 423ms (threshold=20ms); ack: seqno: 8199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1216085, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:39,306 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629935_1099518554565] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 36885 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 495877, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:39,306 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629938_1099518554568] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 515 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1092961, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:47:39,306 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:39,306 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629936_1099518554566] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 8257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 778514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:39,306 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:39,306 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:39,306 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:39,306 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:39,699 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=4054060, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/afb304a5f7f24661ae3e44faa08de473 2015-10-16 12:47:39,705 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/afb304a5f7f24661ae3e44faa08de473 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/afb304a5f7f24661ae3e44faa08de473 2015-10-16 12:47:39,719 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/afb304a5f7f24661ae3e44faa08de473, entries=466290, sequenceid=4054060, filesize=70.8 M 2015-10-16 12:47:39,721 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629935_1099518554565] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 38693 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 566938, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:39,721 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134286800, currentsize=19.39 MB/20332640 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 803ms, sequenceid=4054060, compaction requested=true 2015-10-16 12:47:39,721 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:47:39,721 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 5 compacting, 5 eligible, 10 blocking 2015-10-16 12:47:39,724 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 5 files of size 573340840 starting at candidate #0 after considering 6 permutations with 2 in ratio 2015-10-16 12:47:39,724 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:47:39,724 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:47:39,725 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=546.8 M 2015-10-16 12:47:39,725 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8d18242e35ad47e989925dbf6f3c37ec, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=3740762 2015-10-16 12:47:39,725 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0eb06cec35354048a5a8904f1e2fd289, keycount=1733960, bloomtype=ROW, size=263.4 M, encoding=NONE, seqNum=3914163 2015-10-16 12:47:39,725 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0c986601629c4788b349391937eb0ef5, keycount=466300, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=3960797 2015-10-16 12:47:39,725 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/86f01b04ebf64b2bb870402058dca36a, keycount=466260, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=4007427 2015-10-16 12:47:39,725 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/afb304a5f7f24661ae3e44faa08de473, keycount=466290, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=4054060 2015-10-16 12:47:39,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 324ms (threshold=20ms); ack: seqno: 8368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1675851, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:39,753 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:47:39,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629935_1099518554565] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 38694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 720443, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:39,803 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:47:39,842 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:39,857 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024851808 with entries=78042, filesize=122.34 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024859803 2015-10-16 12:47:39,861 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024841933 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024841933 2015-10-16 12:47:39,862 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629941_1099518554571] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1055170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:39,928 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629941_1099518554571] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 147 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 599330, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:39,929 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:47:39,929 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:47:39,929 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:47:39,929 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:47:40,729 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 846ms (threshold=20ms); ack: seqno: 8535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1270659, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:40,759 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 867ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1657592, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:40,861 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 240 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 772594, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:40,861 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629941_1099518554571] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 4864 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 584616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:40,861 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 8634 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910481, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:40,861 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:47:40,861 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:47:40,861 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:47:40,861 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:47:40,861 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:47:41,010 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 145ms (threshold=20ms); ack: seqno: 8648 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1169059, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:41,180 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 146ms (threshold=20ms); ack: seqno: 8686 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1185460, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:41,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 540ms (threshold=20ms); ack: seqno: 8702 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1170986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:41,787 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:41,787 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:41,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629941_1099518554571] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 9817 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 602675, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:41,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 8730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4984756, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:41,787 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:41,787 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:41,787 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:41,788 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 860ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1039756, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:41,840 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 333 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 51958906, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:41,865 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 8770 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 12307588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:41,917 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 8802 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008704, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:42,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 8847 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910879, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:42,633 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629941_1099518554571] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 14108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 401362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:42,634 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:42,634 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:42,634 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:42,634 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:42,634 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:42,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 698ms (threshold=20ms); ack: seqno: 8871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1383766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:42,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 889ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1230044, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:43,511 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629941_1099518554571] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 18333 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 609160, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:43,511 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:43,511 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:43,511 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:43,511 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:43,511 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:47:43,701 DEBUG [PriorityRpcServer.handler=6,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:47:43,701 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:47:43,701 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:47:43,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 882ms (threshold=20ms); ack: seqno: 9038 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1222839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:43,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 883ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1215159, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:43,860 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629941_1099518554571] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 19607 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 558443, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:44,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629941_1099518554571] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 22479 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 8981195, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:44,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 892ms (threshold=20ms); ack: seqno: 9205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1271191, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:44,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 890ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1228086, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:44,915 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629941_1099518554571] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 24425 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912910, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:44,915 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 9357 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 854843, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:44,915 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 895 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804670, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:44,915 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 81ms (threshold=20ms) 2015-10-16 12:47:44,915 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 80ms (threshold=20ms) 2015-10-16 12:47:44,915 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 80ms (threshold=20ms) 2015-10-16 12:47:44,915 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 80ms (threshold=20ms) 2015-10-16 12:47:44,915 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 80ms (threshold=20ms) 2015-10-16 12:47:45,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629941_1099518554571] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 26644 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 469624, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:45,372 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:47:45,372 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:47:45,372 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:47:45,372 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:47:45,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 804ms (threshold=20ms); ack: seqno: 9372 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1274608, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:45,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 796ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1097092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:46,231 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629941_1099518554571] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 30924 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 611050, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:46,231 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:46,231 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:46,231 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:46,231 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:46,231 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:46,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 873ms (threshold=20ms); ack: seqno: 9539 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1305807, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:46,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 876ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019659, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:47,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629941_1099518554571] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 35188 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 467302, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:47,098 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:47,098 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:47,098 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:47,098 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:47,098 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:47,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 864ms (threshold=20ms); ack: seqno: 9706 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1274181, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:47,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 880ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1032254, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:47,798 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:47:47,842 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629941_1099518554571] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 38951 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 535806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:47,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 1335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1888944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:47,849 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024859803 with entries=77624, filesize=121.69 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024867812 2015-10-16 12:47:47,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629942_1099518554572] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907517, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:48,022 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629942_1099518554572] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 463 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 594893, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:48,022 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:48,022 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:48,022 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:48,022 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:48,022 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:48,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 874ms (threshold=20ms); ack: seqno: 9873 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1344144, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:48,790 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 890ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062169, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:48,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629942_1099518554572] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 5200 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701007, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:48,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 10006 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819865, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:48,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 1566 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 773085, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:48,879 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:48,879 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:48,879 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:48,879 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:48,879 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:49,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 827ms (threshold=20ms); ack: seqno: 10040 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1477629, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:49,790 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629942_1099518554572] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 10119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 466011, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:49,790 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 10073 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 799555, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:49,791 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:49,791 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:49,791 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:49,791 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:49,791 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:49,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 834ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1234621, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:50,593 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629942_1099518554572] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 14466 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 460383, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:50,593 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:50,593 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:50,593 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:50,593 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:50,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 835ms (threshold=20ms); ack: seqno: 10207 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1601429, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:50,803 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 877ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 11122088, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:51,425 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629942_1099518554572] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 18628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 714691, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:51,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629942_1099518554572] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 18637 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 702983, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:51,461 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:51,461 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:51,461 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:51,461 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:51,461 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:51,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 860ms (threshold=20ms); ack: seqno: 10374 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1598154, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:51,793 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629940_1099518554570] hdfs.DFSClient: Slow ReadProcessor read fields took 858ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1146131, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:51,820 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629939_1099518554569] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 10409 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917473, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:47:52,262 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629942_1099518554572] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 22853 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 417543, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:52,262 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 12:47:52,262 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 12:47:52,262 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 12:47:52,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 813ms (threshold=20ms); ack: seqno: 10543 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1491159, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:47:52,797 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629944_1099518554574] hdfs.DFSClient: Slow ReadProcessor read fields took 881ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7316450, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:52,803 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 10594 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971017, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:47:52,898 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 10653 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964396, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:47:52,974 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 10704 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1027622, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:47:53,389 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629942_1099518554572] hdfs.DFSClient: Slow ReadProcessor read fields took 556ms (threshold=20ms); ack: seqno: 26210 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902277, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:53,389 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 557ms (threshold=20ms) 2015-10-16 12:47:53,389 INFO [sync.4] wal.FSHLog: Slow sync cost: 557 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:53,399 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 566ms (threshold=20ms) 2015-10-16 12:47:53,399 INFO [sync.2] wal.FSHLog: Slow sync cost: 566 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:53,399 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 567ms (threshold=20ms) 2015-10-16 12:47:53,399 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 567ms (threshold=20ms) 2015-10-16 12:47:53,399 INFO [sync.3] wal.FSHLog: Slow sync cost: 567 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:53,399 INFO [sync.1] wal.FSHLog: Slow sync cost: 567 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:53,399 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 566ms (threshold=20ms) 2015-10-16 12:47:53,399 INFO [sync.0] wal.FSHLog: Slow sync cost: 566 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:53,585 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629942_1099518554572] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 27060 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 451055, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:53,585 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:47:53,585 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:47:53,585 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:47:53,585 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:47:53,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 752ms (threshold=20ms); ack: seqno: 10710 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1569678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:47:53,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629944_1099518554574] hdfs.DFSClient: Slow ReadProcessor read fields took 884ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1387539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:54,370 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629942_1099518554572] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 31310 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 517235, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:54,370 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:54,370 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:54,370 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:54,370 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:54,370 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:54,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 866ms (threshold=20ms); ack: seqno: 10879 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1340527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:47:54,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629944_1099518554574] hdfs.DFSClient: Slow ReadProcessor read fields took 881ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1311074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:55,148 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629942_1099518554572] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 35495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 474171, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:47:55,148 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:55,148 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:55,148 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:55,148 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:55,148 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:55,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 873ms (threshold=20ms); ack: seqno: 11046 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1259966, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:47:55,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629944_1099518554574] hdfs.DFSClient: Slow ReadProcessor read fields took 882ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1382262, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:55,799 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 11091 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859853, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:47:55,808 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:47:55,845 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:47:55,865 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024867812 with entries=77934, filesize=122.17 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024875808 2015-10-16 12:47:55,868 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062112, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:55,947 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 218 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 514803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:55,947 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629944_1099518554574] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 2835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 815773, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:55,947 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:55,948 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:55,948 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:55,948 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:55,948 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:47:56,315 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 1611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 748108, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:56,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 835ms (threshold=20ms); ack: seqno: 11215 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1313142, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:47:56,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629944_1099518554574] hdfs.DFSClient: Slow ReadProcessor read fields took 840ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1299414, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:56,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 4836 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 500187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:56,963 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:56,963 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:56,963 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:47:56,963 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:56,963 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:47:57,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 825ms (threshold=20ms); ack: seqno: 11216 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825334939, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:47:57,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 11382 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1007079, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:47:57,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 11531 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 693069, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:47:57,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 9258 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 530546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:47:57,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629944_1099518554574] hdfs.DFSClient: Slow ReadProcessor read fields took 963ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 70965597, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:57,874 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:57,874 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:57,874 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:47:57,874 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:47:58,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 844ms (threshold=20ms); ack: seqno: 11549 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1539569, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:47:58,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629944_1099518554574] hdfs.DFSClient: Slow ReadProcessor read fields took 838ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1411293, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:59,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 892ms (threshold=20ms); ack: seqno: 11716 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1588923, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:47:59,786 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 11745 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 806640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:47:59,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629944_1099518554574] hdfs.DFSClient: Slow ReadProcessor read fields took 892ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1271469, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:47:59,909 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629944_1099518554574] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 3465 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 735977, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:48:00,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 857ms (threshold=20ms); ack: seqno: 11883 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1559577, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:00,793 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629944_1099518554574] hdfs.DFSClient: Slow ReadProcessor read fields took 855ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1604393, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:48:00,810 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 11924 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:01,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 816ms (threshold=20ms); ack: seqno: 12050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1454534, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:01,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629944_1099518554574] hdfs.DFSClient: Slow ReadProcessor read fields took 861ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1556158, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:48:02,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 899ms (threshold=20ms); ack: seqno: 12217 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7386245, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:02,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629944_1099518554574] hdfs.DFSClient: Slow ReadProcessor read fields took 886ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1471157, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:48:02,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 12248 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 968514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:02,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 12302 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945515, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:03,008 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 12356 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782137, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:03,018 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#20 3241219/3599080 (90.06%), rate=12619.26 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=20.00 MB/sec, activeCompactions=2] 2015-10-16 12:48:03,024 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#20 sleep 705 ms because current throughput is 33.90 MB/sec, max allowed is 10.00 MB/sec, already slept 72 time(s) and total slept time is 46907 ms till now. 2015-10-16 12:48:03,732 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629943_1099518554573] hdfs.DFSClient: Slow ReadProcessor read fields took 707ms (threshold=20ms); ack: seqno: 12384 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1324826, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:03,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629944_1099518554574] hdfs.DFSClient: Slow ReadProcessor read fields took 885ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1575239, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:48:03,911 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629944_1099518554574] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 4137 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 727342, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:48:04,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 882ms (threshold=20ms); ack: seqno: 12553 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1416363, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:04,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629947_1099518554577] hdfs.DFSClient: Slow ReadProcessor read fields took 830ms (threshold=20ms); ack: seqno: 4181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1468480, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:48:04,964 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 214ms (threshold=20ms); ack: seqno: 12585 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1214123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:05,307 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 309ms (threshold=20ms); ack: seqno: 12641 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1235747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:05,732 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 377ms (threshold=20ms); ack: seqno: 12720 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1476144, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:05,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629947_1099518554577] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 4348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1295103, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:48:05,999 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 220ms (threshold=20ms); ack: seqno: 12800 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1058842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:06,385 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 341ms (threshold=20ms); ack: seqno: 12855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1372886, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:06,732 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 329ms (threshold=20ms); ack: seqno: 12887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1314926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:06,782 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 12921 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888958, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:06,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629947_1099518554577] hdfs.DFSClient: Slow ReadProcessor read fields took 897ms (threshold=20ms); ack: seqno: 4517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1390327, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:48:06,808 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 12927 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 882696, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:06,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 12993 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834378, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:07,732 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 815ms (threshold=20ms); ack: seqno: 13054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1349707, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:07,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629947_1099518554577] hdfs.DFSClient: Slow ReadProcessor read fields took 894ms (threshold=20ms); ack: seqno: 4684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1319388, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:48:07,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629947_1099518554577] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 4820 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 730473, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:48:08,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 886ms (threshold=20ms); ack: seqno: 13221 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1397911, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:08,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629947_1099518554577] hdfs.DFSClient: Slow ReadProcessor read fields took 860ms (threshold=20ms); ack: seqno: 4851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1320320, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:48:09,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 877ms (threshold=20ms); ack: seqno: 13390 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1389841, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:09,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629947_1099518554577] hdfs.DFSClient: Slow ReadProcessor read fields took 894ms (threshold=20ms); ack: seqno: 5018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1358420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:48:10,732 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 870ms (threshold=20ms); ack: seqno: 13559 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1398512, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:10,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629947_1099518554577] hdfs.DFSClient: Slow ReadProcessor read fields took 890ms (threshold=20ms); ack: seqno: 5185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1436980, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:48:11,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 879ms (threshold=20ms); ack: seqno: 13726 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1395167, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:11,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629947_1099518554577] hdfs.DFSClient: Slow ReadProcessor read fields took 878ms (threshold=20ms); ack: seqno: 5352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1506737, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:48:11,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 13780 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 959706, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:12,064 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 200ms (threshold=20ms); ack: seqno: 13831 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 952672, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:12,122 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 13880 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:12,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 600ms (threshold=20ms); ack: seqno: 13893 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1606199, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:12,785 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 13894 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1713081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:12,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629947_1099518554577] hdfs.DFSClient: Slow ReadProcessor read fields took 881ms (threshold=20ms); ack: seqno: 5521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1302840, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:48:12,865 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 13979 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861957, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:12,960 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 14037 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966630, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:13,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 758ms (threshold=20ms); ack: seqno: 14060 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1377287, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:13,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629947_1099518554577] hdfs.DFSClient: Slow ReadProcessor read fields took 869ms (threshold=20ms); ack: seqno: 5688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1517610, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:48:14,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 884ms (threshold=20ms); ack: seqno: 14227 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1378417, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:14,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629947_1099518554577] hdfs.DFSClient: Slow ReadProcessor read fields took 879ms (threshold=20ms); ack: seqno: 5855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1211879, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:48:15,732 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 891ms (threshold=20ms); ack: seqno: 14394 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1401650, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:15,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629947_1099518554577] hdfs.DFSClient: Slow ReadProcessor read fields took 880ms (threshold=20ms); ack: seqno: 6022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1644238, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:48:16,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629946_1099518554576] hdfs.DFSClient: Slow ReadProcessor read fields took 887ms (threshold=20ms); ack: seqno: 14561 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1512070, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:16,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629947_1099518554577] hdfs.DFSClient: Slow ReadProcessor read fields took 877ms (threshold=20ms); ack: seqno: 6189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188273, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:48:17,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629948_1099518554578] hdfs.DFSClient: Slow ReadProcessor read fields took 890ms (threshold=20ms); ack: seqno: 14730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1779468, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:48:17,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629949_1099518554579] hdfs.DFSClient: Slow ReadProcessor read fields took 887ms (threshold=20ms); ack: seqno: 6358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1352765, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:18,732 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629948_1099518554578] hdfs.DFSClient: Slow ReadProcessor read fields took 886ms (threshold=20ms); ack: seqno: 14897 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1775772, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:48:18,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629949_1099518554579] hdfs.DFSClient: Slow ReadProcessor read fields took 894ms (threshold=20ms); ack: seqno: 6525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1811861, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:19,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629948_1099518554578] hdfs.DFSClient: Slow ReadProcessor read fields took 888ms (threshold=20ms); ack: seqno: 15064 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1525503, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:48:19,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629949_1099518554579] hdfs.DFSClient: Slow ReadProcessor read fields took 890ms (threshold=20ms); ack: seqno: 6694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1402551, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:20,732 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629948_1099518554578] hdfs.DFSClient: Slow ReadProcessor read fields took 881ms (threshold=20ms); ack: seqno: 15231 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1494490, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:48:20,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629948_1099518554578] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 15238 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 958905, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:48:20,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629949_1099518554579] hdfs.DFSClient: Slow ReadProcessor read fields took 891ms (threshold=20ms); ack: seqno: 6861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1349113, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:21,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629948_1099518554578] hdfs.DFSClient: Slow ReadProcessor read fields took 855ms (threshold=20ms); ack: seqno: 15398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1411318, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:48:21,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629949_1099518554579] hdfs.DFSClient: Slow ReadProcessor read fields took 894ms (threshold=20ms); ack: seqno: 7028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1590153, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:21,840 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629948_1099518554578] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 15445 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 73748820, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:48:22,734 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629948_1099518554578] hdfs.DFSClient: Slow ReadProcessor read fields took 863ms (threshold=20ms); ack: seqno: 15567 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1567302, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:48:22,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629949_1099518554579] hdfs.DFSClient: Slow ReadProcessor read fields took 892ms (threshold=20ms); ack: seqno: 7195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1404088, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:22,863 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629949_1099518554579] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 7196 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 68217227, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:23,734 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629948_1099518554578] hdfs.DFSClient: Slow ReadProcessor read fields took 885ms (threshold=20ms); ack: seqno: 15734 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1497753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:48:23,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629949_1099518554579] hdfs.DFSClient: Slow ReadProcessor read fields took 874ms (threshold=20ms); ack: seqno: 7362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1391574, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:23,856 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629948_1099518554578] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 15837 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008217, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:48:23,946 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629948_1099518554578] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 15891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 807303, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:48:24,734 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629948_1099518554578] hdfs.DFSClient: Slow ReadProcessor read fields took 781ms (threshold=20ms); ack: seqno: 15901 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1412519, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:48:24,779 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629948_1099518554578] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 15922 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 784252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:48:24,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629949_1099518554579] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 7529 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1384752, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:24,838 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#20 average throughput is 11.69 MB/sec, slept 94 time(s) and total slept time is 64452 ms. 1 active compactions remaining, total limit is 20.00 MB/sec 2015-10-16 12:48:24,862 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/fb8e84ddaef844f78ab1023bd41241e0 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fb8e84ddaef844f78ab1023bd41241e0 2015-10-16 12:48:24,877 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 26546ms (threshold=20ms); ack: seqno: 11620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 788238, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:24,877 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:48:24,881 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:48:24,887 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5eec301127f84273841e8419ac0cecfa, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5eec301127f84273841e8419ac0cecfa 2015-10-16 12:48:24,896 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/580a0b059e6044c282eed9675e296b48, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/580a0b059e6044c282eed9675e296b48 2015-10-16 12:48:24,904 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/27f9fdba80de429dab8ea75581c3b170, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/27f9fdba80de429dab8ea75581c3b170 2015-10-16 12:48:24,912 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8d5cfa97ec3d465da1aa1060993f2b31, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8d5cfa97ec3d465da1aa1060993f2b31 2015-10-16 12:48:24,921 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2620664d2de24dfb94d746171e29f6fa, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2620664d2de24dfb94d746171e29f6fa 2015-10-16 12:48:24,921 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into fb8e84ddaef844f78ab1023bd41241e0(size=983.5 M), total size for store is 6.0 G. This selection was in queue for 0sec, and took 1mins, 21sec to execute. 2015-10-16 12:48:24,921 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=5, fileSize=983.6 M, priority=0, time=36902731837462251; duration=1mins, 21sec 2015-10-16 12:48:24,921 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:48:24,921 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 5 compacting, 0 eligible, 10 blocking 2015-10-16 12:48:24,921 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:48:24,921 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:48:24,921 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:48:25,116 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536924496 > lower bound=16777216 2015-10-16 12:48:25,116 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 12:48:25,218 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 341ms (threshold=20ms); ack: seqno: 11621 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 829880, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:25,296 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629949_1099518554579] hdfs.DFSClient: Slow ReadProcessor read fields took 391ms (threshold=20ms); ack: seqno: 7696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1192641, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:25,796 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629949_1099518554579] hdfs.DFSClient: Slow ReadProcessor read fields took 358ms (threshold=20ms); ack: seqno: 7863 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1295633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:26,296 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629949_1099518554579] hdfs.DFSClient: Slow ReadProcessor read fields took 397ms (threshold=20ms); ack: seqno: 8030 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1328342, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:26,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629949_1099518554579] hdfs.DFSClient: Slow ReadProcessor read fields took 398ms (threshold=20ms); ack: seqno: 8199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1225202, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:27,016 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629949_1099518554579] hdfs.DFSClient: Slow ReadProcessor read fields took 146ms (threshold=20ms); ack: seqno: 8321 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 146257757, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:27,295 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629952_1099518554582] hdfs.DFSClient: Slow ReadProcessor read fields took 246ms (threshold=20ms); ack: seqno: 8369 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1488376, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:48:27,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629951_1099518554581] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 3532 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 958485, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:48:27,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629952_1099518554582] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 8515 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 820135, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:48:27,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629952_1099518554582] hdfs.DFSClient: Slow ReadProcessor read fields took 375ms (threshold=20ms); ack: seqno: 8535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1186030, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:48:28,046 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=4240502, memsize=512.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a11f78abbe6848f09c630fb9e406e9f2 2015-10-16 12:48:28,053 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a11f78abbe6848f09c630fb9e406e9f2 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a11f78abbe6848f09c630fb9e406e9f2 2015-10-16 12:48:28,066 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a11f78abbe6848f09c630fb9e406e9f2, entries=1864370, sequenceid=4240502, filesize=283.2 M 2015-10-16 12:48:28,182 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 2963ms (threshold=20ms); ack: seqno: 11622 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 114429348, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:28,182 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 115ms (threshold=20ms) 2015-10-16 12:48:28,182 INFO [sync.4] wal.FSHLog: Slow sync cost: 115 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:28,182 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536924080, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 3066ms, sequenceid=4240502, compaction requested=true 2015-10-16 12:48:28,182 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:48:28,182 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 5 compacting, 1 eligible, 10 blocking 2015-10-16 12:48:28,183 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:48:28,183 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:48:28,183 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:48:28,296 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629952_1099518554582] hdfs.DFSClient: Slow ReadProcessor read fields took 385ms (threshold=20ms); ack: seqno: 8702 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1519996, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:48:28,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629952_1099518554582] hdfs.DFSClient: Slow ReadProcessor read fields took 394ms (threshold=20ms); ack: seqno: 8871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1588108, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:48:28,811 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#21 average throughput is 10.84 MB/sec, slept 53 time(s) and total slept time is 42400 ms. 0 active compactions remaining, total limit is 20.00 MB/sec 2015-10-16 12:48:28,828 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a7ab7d2869bf43c2b55cd843cdea0421 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a7ab7d2869bf43c2b55cd843cdea0421 2015-10-16 12:48:28,843 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 662ms (threshold=20ms); ack: seqno: 11623 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790328, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:28,844 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:48:28,848 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:48:28,854 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8d18242e35ad47e989925dbf6f3c37ec, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8d18242e35ad47e989925dbf6f3c37ec 2015-10-16 12:48:28,862 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0eb06cec35354048a5a8904f1e2fd289, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0eb06cec35354048a5a8904f1e2fd289 2015-10-16 12:48:28,870 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0c986601629c4788b349391937eb0ef5, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0c986601629c4788b349391937eb0ef5 2015-10-16 12:48:28,879 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/86f01b04ebf64b2bb870402058dca36a, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/86f01b04ebf64b2bb870402058dca36a 2015-10-16 12:48:28,887 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/afb304a5f7f24661ae3e44faa08de473, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/afb304a5f7f24661ae3e44faa08de473 2015-10-16 12:48:28,887 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into a7ab7d2869bf43c2b55cd843cdea0421(size=546.7 M), total size for store is 6.3 G. This selection was in queue for 0sec, and took 49sec to execute. 2015-10-16 12:48:28,887 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=5, fileSize=546.8 M, priority=-1, time=36902768578533492; duration=49sec 2015-10-16 12:48:28,887 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:48:36,507 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 7663ms (threshold=20ms); ack: seqno: 11624 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 952037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:36,799 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 212ms (threshold=20ms); ack: seqno: 11705 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 824842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:37,010 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 209ms (threshold=20ms); ack: seqno: 11712 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 848384, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:37,223 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 211ms (threshold=20ms); ack: seqno: 11727 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 872705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:37,279 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 11728 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808648, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:37,502 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 220ms (threshold=20ms); ack: seqno: 11735 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868603, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:37,554 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 11736 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811402, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:37,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 210ms (threshold=20ms); ack: seqno: 11741 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863132, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:37,923 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 157ms (threshold=20ms); ack: seqno: 11746 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 860642, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:38,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 195ms (threshold=20ms); ack: seqno: 11747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 855387, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:38,273 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 154ms (threshold=20ms); ack: seqno: 11752 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 831296, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:38,379 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 11758 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941750, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:38,485 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 11762 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813439, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:38,541 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 11766 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 852416, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:38,603 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 11780 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 776061, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:38,662 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 11785 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 772391, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:38,724 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 11793 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 789629, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:38,791 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 11803 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874315, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.14285714285714285, tune compaction throughput to 11.43 MB/sec 2015-10-16 12:48:38,918 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 11806 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:38,978 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 11808 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729906, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:39,034 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 11816 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790651, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:39,141 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 11820 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 759054, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:39,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 209ms (threshold=20ms); ack: seqno: 11838 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841879, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:39,564 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 208ms (threshold=20ms); ack: seqno: 11849 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 822760, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:39,779 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 208ms (threshold=20ms); ack: seqno: 11877 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:39,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 212ms (threshold=20ms); ack: seqno: 11886 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:40,209 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 217ms (threshold=20ms); ack: seqno: 11890 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 891347, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:40,588 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 379ms (threshold=20ms); ack: seqno: 11892 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863142, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:40,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 155ms (threshold=20ms); ack: seqno: 11893 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 843565, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:41,542 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 16182 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 35214139, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:41,542 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:48:41,542 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:48:41,542 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:48:41,542 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:48:41,543 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:48:42,090 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 19066 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 607135, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:42,090 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:48:42,090 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:48:42,090 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:48:42,090 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:48:42,090 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:48:43,038 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 24586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 474854, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:43,038 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:43,038 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:43,038 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:43,038 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:48:43,038 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:43,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 28350 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 490146, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:44,130 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 30193 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 599780, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:44,131 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:48:44,131 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:48:44,994 DEBUG [PriorityRpcServer.handler=2,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:48:44,994 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134272976 > lower bound=16777216 2015-10-16 12:48:44,994 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:48:45,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 35116 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 542408, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:45,145 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629954_1099518554584] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 101 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847137, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:45,145 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629945_1099518554575] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 35564 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 617931, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:48:45,145 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:45,145 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:45,145 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:45,145 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:45,145 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:45,778 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:48:45,811 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024875808 with entries=77837, filesize=122.01 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024925778 2015-10-16 12:48:45,813 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024851808 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024851808 2015-10-16 12:48:45,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629955_1099518554585] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 17 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 596255, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:45,846 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:48:45,846 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:48:45,846 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:48:45,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629954_1099518554584] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 1070 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 753037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:45,846 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:48:45,849 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024859803 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024859803 2015-10-16 12:48:45,853 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024867812 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024867812 2015-10-16 12:48:45,911 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=4287132, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a1cbf15689154134b82f6dc5cd6572c6 2015-10-16 12:48:45,917 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a1cbf15689154134b82f6dc5cd6572c6 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a1cbf15689154134b82f6dc5cd6572c6 2015-10-16 12:48:45,923 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a1cbf15689154134b82f6dc5cd6572c6, entries=466250, sequenceid=4287132, filesize=70.8 M 2015-10-16 12:48:45,924 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134275440, currentsize=25.03 MB/26247520 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 930ms, sequenceid=4287132, compaction requested=true 2015-10-16 12:48:45,924 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:48:45,924 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 5 store files, 0 compacting, 5 eligible, 10 blocking 2015-10-16 12:48:45,926 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 1975792232 starting at candidate #1 after considering 6 permutations with 2 in ratio 2015-10-16 12:48:45,926 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:48:45,926 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:48:45,926 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=1.8 G 2015-10-16 12:48:45,926 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fb8e84ddaef844f78ab1023bd41241e0, keycount=6474650, bloomtype=ROW, size=983.5 M, encoding=NONE, seqNum=3694131 2015-10-16 12:48:45,926 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a7ab7d2869bf43c2b55cd843cdea0421, keycount=3599080, bloomtype=ROW, size=546.7 M, encoding=NONE, seqNum=4054060 2015-10-16 12:48:45,927 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a11f78abbe6848f09c630fb9e406e9f2, keycount=1864370, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=4240502 2015-10-16 12:48:45,927 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a1cbf15689154134b82f6dc5cd6572c6, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=4287132 2015-10-16 12:48:45,948 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:48:46,756 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:48:46,756 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:48:46,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629955_1099518554585] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 4634 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 624085, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:46,830 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629956_1099518554586] hdfs.DFSClient: Slow ReadProcessor read fields took 734ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1215342, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:48:47,631 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629955_1099518554585] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 9422 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 586773, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:47,631 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:48:47,631 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:47,631 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:48:47,631 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:48:47,631 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:48:47,705 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629956_1099518554586] hdfs.DFSClient: Slow ReadProcessor read fields took 735ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1306668, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:48:48,546 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:48:48,546 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:48:48,547 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629955_1099518554585] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 14328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489999, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:48,546 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:48:48,547 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:48:48,580 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629956_1099518554586] hdfs.DFSClient: Slow ReadProcessor read fields took 735ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1246917, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:48:48,754 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629955_1099518554585] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 14816 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 817888, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:49,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629956_1099518554586] hdfs.DFSClient: Slow ReadProcessor read fields took 765ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1272936, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:48:49,517 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629956_1099518554586] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 701 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 748875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:48:49,517 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629955_1099518554585] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 19050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 394190, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:49,517 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:48:49,517 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:48:49,517 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:48:49,517 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:48:49,517 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:48:49,562 DEBUG [PriorityRpcServer.handler=9,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:48:49,563 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134269888 > lower bound=16777216 2015-10-16 12:48:49,563 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:48:50,180 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629955_1099518554585] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 22329 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 587181, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:50,180 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629957_1099518554587] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 882 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 742190, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:48:50,180 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:48:50,180 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:48:50,180 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:48:50,180 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:48:50,180 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:48:50,330 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629956_1099518554586] hdfs.DFSClient: Slow ReadProcessor read fields took 723ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1301183, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:48:50,377 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=4333766, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ada22472b4bd4f0fbf83dd18a6e31b9c 2015-10-16 12:48:50,383 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ada22472b4bd4f0fbf83dd18a6e31b9c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ada22472b4bd4f0fbf83dd18a6e31b9c 2015-10-16 12:48:50,415 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ada22472b4bd4f0fbf83dd18a6e31b9c, entries=466300, sequenceid=4333766, filesize=70.8 M 2015-10-16 12:48:50,417 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134289920, currentsize=23.72 MB/24868480 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 853ms, sequenceid=4333766, compaction requested=false 2015-10-16 12:48:50,989 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629955_1099518554585] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 26721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 445586, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:50,989 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:50,990 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:50,990 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:50,990 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:48:50,990 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:51,221 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629956_1099518554586] hdfs.DFSClient: Slow ReadProcessor read fields took 764ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 17354286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:48:51,889 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629955_1099518554585] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 31582 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 445903, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:51,889 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:48:51,889 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:48:51,889 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:48:51,889 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:51,889 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:52,080 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629956_1099518554586] hdfs.DFSClient: Slow ReadProcessor read fields took 757ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:48:52,772 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629955_1099518554585] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 36415 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 427598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:48:52,772 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:48:52,773 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:52,773 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:52,773 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:52,773 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:52,955 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629956_1099518554586] hdfs.DFSClient: Slow ReadProcessor read fields took 757ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1259155, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:48:53,248 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:48:53,286 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024925778 with entries=77857, filesize=122.05 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024933249 2015-10-16 12:48:53,288 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024875808 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024875808 2015-10-16 12:48:53,289 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629958_1099518554588] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1071460, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:53,436 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629958_1099518554588] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 214 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 430770, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:53,803 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629958_1099518554588] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 2156 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 497767, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:53,803 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:48:53,803 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:48:53,803 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:48:53,803 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:48:53,803 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:48:53,830 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629956_1099518554586] hdfs.DFSClient: Slow ReadProcessor read fields took 752ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1185968, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:48:54,044 DEBUG [PriorityRpcServer.handler=7,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:48:54,044 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134266016 > lower bound=16777216 2015-10-16 12:48:54,044 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:48:54,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629958_1099518554588] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 5987 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 595889, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:54,568 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 101ms (threshold=20ms) 2015-10-16 12:48:54,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629959_1099518554589] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 690023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:48:54,569 INFO [sync.2] wal.FSHLog: Slow sync cost: 101 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:54,569 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 101ms (threshold=20ms) 2015-10-16 12:48:54,569 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 101ms (threshold=20ms) 2015-10-16 12:48:54,568 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 101ms (threshold=20ms) 2015-10-16 12:48:54,568 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 101ms (threshold=20ms) 2015-10-16 12:48:54,569 INFO [sync.3] wal.FSHLog: Slow sync cost: 101 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:54,569 INFO [sync.0] wal.FSHLog: Slow sync cost: 101 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:54,569 INFO [sync.4] wal.FSHLog: Slow sync cost: 101 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:54,569 INFO [sync.1] wal.FSHLog: Slow sync cost: 101 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:54,707 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629956_1099518554586] hdfs.DFSClient: Slow ReadProcessor read fields took 755ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1206286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:48:54,919 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=4380393, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a59a0ab0c1094771836206b258e67f63 2015-10-16 12:48:54,924 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a59a0ab0c1094771836206b258e67f63 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a59a0ab0c1094771836206b258e67f63 2015-10-16 12:48:54,931 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a59a0ab0c1094771836206b258e67f63, entries=466230, sequenceid=4380393, filesize=70.8 M 2015-10-16 12:48:54,932 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134271360, currentsize=24.33 MB/25515760 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 888ms, sequenceid=4380393, compaction requested=true 2015-10-16 12:48:54,932 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:48:54,932 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 4 compacting, 2 eligible, 10 blocking 2015-10-16 12:48:54,933 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:48:54,934 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:48:54,934 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:48:55,209 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629958_1099518554588] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 9622 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 462783, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:55,209 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:48:55,209 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:48:55,209 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:48:55,209 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:48:55,210 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:48:55,581 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629956_1099518554586] hdfs.DFSClient: Slow ReadProcessor read fields took 746ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066683, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:48:56,038 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629958_1099518554588] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 14409 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 461738, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:56,038 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:48:56,038 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:56,038 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:48:56,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629956_1099518554586] hdfs.DFSClient: Slow ReadProcessor read fields took 754ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1260764, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:48:56,882 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629958_1099518554588] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 19347 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 428156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:56,882 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:56,882 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:56,882 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:56,882 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:56,882 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:57,331 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629960_1099518554590] hdfs.DFSClient: Slow ReadProcessor read fields took 631ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1389133, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:48:57,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629958_1099518554588] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 23789 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 494239, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:57,825 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:57,825 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629958_1099518554588] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 24127 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863844, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:57,825 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:57,825 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:57,825 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:48:57,825 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:48:58,206 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629960_1099518554590] hdfs.DFSClient: Slow ReadProcessor read fields took 743ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1227884, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:48:58,348 DEBUG [PriorityRpcServer.handler=1,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:48:58,349 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134270176 > lower bound=16777216 2015-10-16 12:48:58,349 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:48:58,373 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629958_1099518554588] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 27137 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 560488, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:58,621 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629961_1099518554591] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 345 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:48:58,621 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629958_1099518554588] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 28314 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 473003, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:58,622 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:58,622 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:58,622 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:58,622 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:58,622 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:48:59,081 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629960_1099518554590] hdfs.DFSClient: Slow ReadProcessor read fields took 750ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1224689, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:48:59,152 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=4427027, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c5c3d756f50f4b04a5274c53deb515a2 2015-10-16 12:48:59,158 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c5c3d756f50f4b04a5274c53deb515a2 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c5c3d756f50f4b04a5274c53deb515a2 2015-10-16 12:48:59,164 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c5c3d756f50f4b04a5274c53deb515a2, entries=466300, sequenceid=4427027, filesize=70.8 M 2015-10-16 12:48:59,165 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134289920, currentsize=23.63 MB/24778720 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 816ms, sequenceid=4427027, compaction requested=true 2015-10-16 12:48:59,165 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:48:59,166 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 4 compacting, 3 eligible, 10 blocking 2015-10-16 12:48:59,167 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222830280 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 12:48:59,167 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:48:59,167 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:48:59,167 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 12:48:59,167 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ada22472b4bd4f0fbf83dd18a6e31b9c, keycount=466300, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=4333766 2015-10-16 12:48:59,167 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a59a0ab0c1094771836206b258e67f63, keycount=466230, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=4380393 2015-10-16 12:48:59,167 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c5c3d756f50f4b04a5274c53deb515a2, keycount=466300, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=4427027 2015-10-16 12:48:59,182 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:48:59,229 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629960_1099518554590] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 2641 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012240, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:48:59,229 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629958_1099518554588] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 31599 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 451970, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:48:59,229 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629962_1099518554592] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 0 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2273213, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:48:59,229 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:48:59,229 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:48:59,229 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:48:59,229 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:48:59,229 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:49:00,039 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:49:00,040 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:49:00,040 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:49:00,039 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:49:00,039 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:49:00,485 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:49:00,519 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024933249 with entries=77881, filesize=122.09 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024940485 2015-10-16 12:49:00,520 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024925778 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024925778 2015-10-16 12:49:00,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629960_1099518554590] hdfs.DFSClient: Slow ReadProcessor read fields took 1576ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1331136, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:00,939 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629963_1099518554593] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 2185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 507262, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:49:00,939 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629960_1099518554590] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 2787 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 876479, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:00,939 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:00,939 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:00,939 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:49:00,939 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:00,939 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:00,940 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629962_1099518554592] hdfs.DFSClient: Slow ReadProcessor read fields took 1574ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1350810, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:01,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629963_1099518554593] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 6881 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 525586, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:49:01,787 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:01,787 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:01,787 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:01,787 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:01,787 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:01,909 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629963_1099518554593] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 6965 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870178, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:49:02,075 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629963_1099518554593] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 7728 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 29933494, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:49:02,075 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:02,075 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:02,125 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629963_1099518554593] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 7730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 79674767, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:49:02,125 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 79ms (threshold=20ms) 2015-10-16 12:49:02,125 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 80ms (threshold=20ms) 2015-10-16 12:49:02,125 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 79ms (threshold=20ms) 2015-10-16 12:49:02,125 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:49:02,125 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:49:02,581 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629960_1099518554590] hdfs.DFSClient: Slow ReadProcessor read fields took 1602ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1286996, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:02,691 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629962_1099518554592] hdfs.DFSClient: Slow ReadProcessor read fields took 1593ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1786592, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:02,747 DEBUG [PriorityRpcServer.handler=9,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:49:02,748 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134283472 > lower bound=16777216 2015-10-16 12:49:02,748 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:49:02,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629962_1099518554592] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 727912, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:02,806 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629963_1099518554593] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 11359 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866320, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:49:03,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629963_1099518554593] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 14738 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 710120, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:49:03,446 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:49:03,446 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:49:03,446 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:49:03,446 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:49:03,446 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:49:03,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629964_1099518554594] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 940 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 688156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:03,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=4473661, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/87bb3be33dc644ca8b7053ae20aeb63e 2015-10-16 12:49:03,599 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/87bb3be33dc644ca8b7053ae20aeb63e as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/87bb3be33dc644ca8b7053ae20aeb63e 2015-10-16 12:49:03,605 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/87bb3be33dc644ca8b7053ae20aeb63e, entries=466300, sequenceid=4473661, filesize=70.8 M 2015-10-16 12:49:03,607 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134291120, currentsize=23.17 MB/24298000 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 858ms, sequenceid=4473661, compaction requested=false 2015-10-16 12:49:04,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629963_1099518554593] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 19840 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 586393, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:49:04,355 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:49:04,355 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:49:04,355 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:49:04,355 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:49:04,355 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:49:04,357 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629960_1099518554590] hdfs.DFSClient: Slow ReadProcessor read fields took 1656ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1329713, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:04,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629962_1099518554592] hdfs.DFSClient: Slow ReadProcessor read fields took 1572ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1318855, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:05,126 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629963_1099518554593] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 24039 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 413706, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:49:05,127 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:05,127 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:05,127 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:05,127 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:05,127 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:06,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629963_1099518554593] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 28511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 606356, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:49:06,108 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629960_1099518554590] hdfs.DFSClient: Slow ReadProcessor read fields took 1630ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1301760, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:06,189 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629963_1099518554593] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 29340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 628273, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:49:06,189 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629960_1099518554590] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 3243 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 752611, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:06,190 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:06,190 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:06,190 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:06,190 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:06,190 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:06,192 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629962_1099518554592] hdfs.DFSClient: Slow ReadProcessor read fields took 1596ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1493051, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:06,990 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629963_1099518554593] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 33796 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 429709, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:49:06,990 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:06,990 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:49:06,990 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:49:06,990 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:49:07,140 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629963_1099518554593] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 34496 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 452734, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:49:07,198 DEBUG [PriorityRpcServer.handler=12,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:49:07,198 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134270496 > lower bound=16777216 2015-10-16 12:49:07,198 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:49:07,224 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629963_1099518554593] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 34847 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 476775, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:49:07,760 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629965_1099518554595] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 781 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948639, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:07,760 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629963_1099518554593] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 37773 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 443273, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:49:07,760 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:07,870 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629960_1099518554590] hdfs.DFSClient: Slow ReadProcessor read fields took 1602ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 13610850, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:07,943 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629962_1099518554592] hdfs.DFSClient: Slow ReadProcessor read fields took 1591ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1894644, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:07,989 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:49:08,033 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:49:08,051 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=4520290, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a5a9ed6be16e40758be730a06fdc5592 2015-10-16 12:49:08,056 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a5a9ed6be16e40758be730a06fdc5592 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a5a9ed6be16e40758be730a06fdc5592 2015-10-16 12:49:08,064 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024940485 with entries=78107, filesize=122.44 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024947989 2015-10-16 12:49:08,067 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024933249 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024933249 2015-10-16 12:49:08,068 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629966_1099518554596] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102836, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:08,073 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a5a9ed6be16e40758be730a06fdc5592, entries=466250, sequenceid=4520290, filesize=70.8 M 2015-10-16 12:49:08,075 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134275840, currentsize=24.72 MB/25924960 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 877ms, sequenceid=4520290, compaction requested=true 2015-10-16 12:49:08,075 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:49:08,413 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:49:08,413 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:49:08,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629966_1099518554596] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 1909 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 359463, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:08,413 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:49:08,413 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:49:08,413 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:49:09,380 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629966_1099518554596] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 7492 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 417284, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:09,380 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:09,380 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:09,380 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:09,380 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:09,380 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:09,607 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629960_1099518554590] hdfs.DFSClient: Slow ReadProcessor read fields took 1602ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1245774, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:09,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629962_1099518554592] hdfs.DFSClient: Slow ReadProcessor read fields took 1587ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1479319, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:09,891 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629966_1099518554596] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 9818 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728189, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:10,232 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629966_1099518554596] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 11688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 408387, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:10,232 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:10,232 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:10,232 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:11,270 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629966_1099518554596] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 17296 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 589160, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:11,270 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:11,270 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:11,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629960_1099518554590] hdfs.DFSClient: Slow ReadProcessor read fields took 1633ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1385165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:11,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629962_1099518554592] hdfs.DFSClient: Slow ReadProcessor read fields took 1630ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1354753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:11,610 DEBUG [PriorityRpcServer.handler=15,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:49:11,610 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134273456 > lower bound=16777216 2015-10-16 12:49:11,610 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:49:11,949 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629967_1099518554597] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 380 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792627, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:11,949 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629966_1099518554596] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 20689 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 556799, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:11,949 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:49:11,949 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:49:11,949 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:49:11,949 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:49:11,949 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:49:12,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=4566918, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/740dc9f44e214ffd83d6ca72fd54f082 2015-10-16 12:49:12,507 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/740dc9f44e214ffd83d6ca72fd54f082 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/740dc9f44e214ffd83d6ca72fd54f082 2015-10-16 12:49:12,513 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/740dc9f44e214ffd83d6ca72fd54f082, entries=466240, sequenceid=4566918, filesize=70.8 M 2015-10-16 12:49:12,515 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134273040, currentsize=26.48 MB/27764880 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 905ms, sequenceid=4566918, compaction requested=true 2015-10-16 12:49:12,515 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 12:49:12,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629966_1099518554596] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 24722 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 452818, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:12,673 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:49:12,673 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:49:12,673 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:49:12,673 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:49:12,673 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:49:13,107 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629960_1099518554590] hdfs.DFSClient: Slow ReadProcessor read fields took 1626ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1202433, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:13,192 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629962_1099518554592] hdfs.DFSClient: Slow ReadProcessor read fields took 1628ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1398591, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:13,421 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629966_1099518554596] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 28920 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 535362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:13,422 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:49:13,422 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:49:13,422 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:49:13,422 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:49:13,422 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:49:13,694 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629966_1099518554596] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 30372 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 431221, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:14,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629966_1099518554596] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 34507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 622034, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:14,430 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:49:14,430 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:14,430 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:49:14,430 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:14,586 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629966_1099518554596] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 34750 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 560050, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:14,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629960_1099518554590] hdfs.DFSClient: Slow ReadProcessor read fields took 1628ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101122, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:14,943 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629962_1099518554592] hdfs.DFSClient: Slow ReadProcessor read fields took 1624ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1416763, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:15,324 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629966_1099518554596] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 38747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 600675, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:15,324 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:49:15,324 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:49:15,324 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:15,324 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:15,409 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:49:15,451 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024947989 with entries=77939, filesize=122.18 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024955410 2015-10-16 12:49:15,453 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024940485 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024940485 2015-10-16 12:49:15,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629969_1099518554599] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111697, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:49:15,995 DEBUG [PriorityRpcServer.handler=15,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:49:15,995 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:49:15,995 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:3), split_queue=0, merge_queue=0 2015-10-16 12:49:16,307 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:49:16,307 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:49:16,308 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629969_1099518554599] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 4882 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 376777, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:49:16,308 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:16,308 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:16,308 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:16,607 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 1620ms (threshold=20ms); ack: seqno: 4182 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1156153, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:16,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629962_1099518554592] hdfs.DFSClient: Slow ReadProcessor read fields took 1628ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1249711, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:17,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629969_1099518554599] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 9068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 441723, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:49:17,063 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:17,063 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:17,063 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:17,963 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:17,963 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:17,963 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:17,963 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:17,963 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:18,356 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 1639ms (threshold=20ms); ack: seqno: 4348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1055038, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:18,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629962_1099518554592] hdfs.DFSClient: Slow ReadProcessor read fields took 1631ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1289060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:18,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629969_1099518554599] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 18769 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 428057, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:49:18,745 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:18,745 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:18,745 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:18,745 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:19,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629969_1099518554599] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 24317 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 403669, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:49:19,746 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:19,746 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:19,746 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:19,746 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:19,747 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:20,053 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629969_1099518554599] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 25444 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 553195, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:49:20,106 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 1633ms (threshold=20ms); ack: seqno: 4517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1018878, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:20,193 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629962_1099518554592] hdfs.DFSClient: Slow ReadProcessor read fields took 1627ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1291215, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:20,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629969_1099518554599] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 28552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 534613, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:49:20,685 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:20,685 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:49:20,685 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:49:20,685 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:49:20,685 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:21,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629969_1099518554599] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 34166 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 522159, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:49:21,745 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:21,745 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:21,745 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:21,745 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:21,745 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:21,857 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 1635ms (threshold=20ms); ack: seqno: 4684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1163613, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:21,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629970_1099518554600] hdfs.DFSClient: Slow ReadProcessor read fields took 1592ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1236377, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:22,569 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:22,569 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:22,570 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629969_1099518554599] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 38423 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 802463, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:49:22,724 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:49:22,759 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024955410 with entries=77869, filesize=122.07 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024962724 2015-10-16 12:49:23,585 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 4597 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 516574, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:23,586 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:23,586 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:23,586 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:23,586 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:23,586 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:23,607 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 1628ms (threshold=20ms); ack: seqno: 4851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1485272, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:23,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629970_1099518554600] hdfs.DFSClient: Slow ReadProcessor read fields took 1607ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207134, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:24,418 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 8740 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 498880, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:24,418 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:49:24,418 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:49:24,418 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:49:25,367 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 1633ms (threshold=20ms); ack: seqno: 5018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 10607532, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:25,419 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 14211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 585009, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:25,420 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 5058 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 725625, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:25,420 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:25,420 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:25,420 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:49:25,420 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:25,420 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:25,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629970_1099518554600] hdfs.DFSClient: Slow ReadProcessor read fields took 1605ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1456877, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:25,633 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 14737 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 788401, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:25,813 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629970_1099518554600] hdfs.DFSClient: Slow ReadProcessor read fields took 371ms (threshold=20ms); ack: seqno: 2508 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 370109332, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:26,381 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 18516 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 539266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:26,381 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:26,381 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:26,381 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:26,381 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:26,381 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:49:27,108 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 1590ms (threshold=20ms); ack: seqno: 5185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188219, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:27,192 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629970_1099518554600] hdfs.DFSClient: Slow ReadProcessor read fields took 1311ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111055, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:27,256 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629970_1099518554600] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 2714 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 867796, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:27,256 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 23311 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 515003, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:27,256 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:27,256 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:27,256 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 5340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790285, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:27,256 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:27,256 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:27,257 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:28,187 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 28349 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 575410, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:28,188 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:28,188 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:28,188 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:28,188 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:28,188 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:49:28,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 1593ms (threshold=20ms); ack: seqno: 5352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1192120, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:28,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629970_1099518554600] hdfs.DFSClient: Slow ReadProcessor read fields took 1591ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1319265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:29,048 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:49:29,048 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:49:29,048 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:49:29,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 5495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 929955, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:29,048 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:49:29,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629970_1099518554600] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 2930 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927830, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:29,048 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:49:30,609 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 1530ms (threshold=20ms); ack: seqno: 5521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1286258, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:30,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629970_1099518554600] hdfs.DFSClient: Slow ReadProcessor read fields took 1585ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1202361, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:32,126 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=689, evicted=0, evictedPerRun=0.0 2015-10-16 12:49:32,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 1641ms (threshold=20ms); ack: seqno: 5688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2020630, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:32,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629970_1099518554600] hdfs.DFSClient: Slow ReadProcessor read fields took 1646ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1365776, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:34,109 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 1649ms (threshold=20ms); ack: seqno: 5855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1770624, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:34,226 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 5974 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 12637109, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:34,226 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629970_1099518554600] hdfs.DFSClient: Slow ReadProcessor read fields took 1680ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1212576, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:49:34,291 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#23 average throughput is 5.89 MB/sec, slept 20 time(s) and total slept time is 31843 ms. 1 active compactions remaining, total limit is 11.43 MB/sec 2015-10-16 12:49:34,316 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1dff2ad0156b4f9d892326753ed03bec as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1dff2ad0156b4f9d892326753ed03bec 2015-10-16 12:49:34,330 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 5003ms (threshold=20ms); ack: seqno: 34632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 625897, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:34,330 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:49:34,334 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:49:34,341 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ada22472b4bd4f0fbf83dd18a6e31b9c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ada22472b4bd4f0fbf83dd18a6e31b9c 2015-10-16 12:49:34,350 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a59a0ab0c1094771836206b258e67f63, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a59a0ab0c1094771836206b258e67f63 2015-10-16 12:49:34,358 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c5c3d756f50f4b04a5274c53deb515a2, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c5c3d756f50f4b04a5274c53deb515a2 2015-10-16 12:49:34,358 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 1dff2ad0156b4f9d892326753ed03bec(size=212.5 M), total size for store is 6.8 G. This selection was in queue for 0sec, and took 35sec to execute. 2015-10-16 12:49:34,358 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=2, time=36902848021036938; duration=35sec 2015-10-16 12:49:34,358 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:3), split_queue=0, merge_queue=0 2015-10-16 12:49:34,358 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 4 compacting, 4 eligible, 10 blocking 2015-10-16 12:49:34,361 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 445655431 starting at candidate #0 after considering 3 permutations with 2 in ratio 2015-10-16 12:49:34,361 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:49:34,361 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:49:34,361 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=425.0 M 2015-10-16 12:49:34,361 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1dff2ad0156b4f9d892326753ed03bec, keycount=1398830, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=4427027 2015-10-16 12:49:34,361 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/87bb3be33dc644ca8b7053ae20aeb63e, keycount=466300, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=4473661 2015-10-16 12:49:34,361 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a5a9ed6be16e40758be730a06fdc5592, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=4520290 2015-10-16 12:49:34,361 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/740dc9f44e214ffd83d6ca72fd54f082, keycount=466240, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=4566918 2015-10-16 12:49:34,383 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:49:34,897 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536923056 > lower bound=16777216 2015-10-16 12:49:34,897 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 12:49:35,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 672ms (threshold=20ms); ack: seqno: 34633 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 796706, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:35,859 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 1607ms (threshold=20ms); ack: seqno: 6022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1267635, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:36,157 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629972_1099518554602] hdfs.DFSClient: Slow ReadProcessor read fields took 1662ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1245698, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:49:36,634 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629974_1099518554604] hdfs.DFSClient: Slow ReadProcessor read fields took 339ms (threshold=20ms); ack: seqno: 2083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 338675556, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=253110, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 12:49:37,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629968_1099518554598] hdfs.DFSClient: Slow ReadProcessor read fields took 1646ms (threshold=20ms); ack: seqno: 6189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1090384, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:49:37,907 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629972_1099518554602] hdfs.DFSClient: Slow ReadProcessor read fields took 1647ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1149128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:49:38,050 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=4753360, memsize=512.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d5cb1dd6386b41e3a42ea2835054ad88 2015-10-16 12:49:38,057 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d5cb1dd6386b41e3a42ea2835054ad88 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d5cb1dd6386b41e3a42ea2835054ad88 2015-10-16 12:49:38,070 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d5cb1dd6386b41e3a42ea2835054ad88, entries=1864370, sequenceid=4753360, filesize=283.2 M 2015-10-16 12:49:38,072 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 3070ms (threshold=20ms); ack: seqno: 34634 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 669806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:38,072 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536922640, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 3175ms, sequenceid=4753360, compaction requested=false 2015-10-16 12:49:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.0, tune compaction throughput to 20.00 MB/sec 2015-10-16 12:49:39,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629975_1099518554605] hdfs.DFSClient: Slow ReadProcessor read fields took 1627ms (threshold=20ms); ack: seqno: 6358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1603652, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:39,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629972_1099518554602] hdfs.DFSClient: Slow ReadProcessor read fields took 1638ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1876053, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:49:40,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629975_1099518554605] hdfs.DFSClient: Slow ReadProcessor read fields took 891ms (threshold=20ms); ack: seqno: 6525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1610803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:40,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629972_1099518554602] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1499021, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:49:41,367 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629975_1099518554605] hdfs.DFSClient: Slow ReadProcessor read fields took 897ms (threshold=20ms); ack: seqno: 6694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 9316119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:41,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629972_1099518554602] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1458875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:49:41,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629972_1099518554602] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 925 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 700015, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:49:42,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629975_1099518554605] hdfs.DFSClient: Slow ReadProcessor read fields took 892ms (threshold=20ms); ack: seqno: 6861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1569562, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:42,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629972_1099518554602] hdfs.DFSClient: Slow ReadProcessor read fields took 827ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1465229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:49:43,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629975_1099518554605] hdfs.DFSClient: Slow ReadProcessor read fields took 900ms (threshold=20ms); ack: seqno: 7028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1555587, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:43,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629972_1099518554602] hdfs.DFSClient: Slow ReadProcessor read fields took 899ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1590876, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:49:44,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629975_1099518554605] hdfs.DFSClient: Slow ReadProcessor read fields took 896ms (threshold=20ms); ack: seqno: 7195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1651143, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:44,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629972_1099518554602] hdfs.DFSClient: Slow ReadProcessor read fields took 899ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1357036, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:49:45,361 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629975_1099518554605] hdfs.DFSClient: Slow ReadProcessor read fields took 901ms (threshold=20ms); ack: seqno: 7362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1504931, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:45,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629972_1099518554602] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1461784, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:49:45,762 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629972_1099518554602] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 1622 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 655089, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:49:46,359 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#22 1286286/2797620 (45.98%), rate=7628.47 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=20.00 MB/sec, activeCompactions=2] 2015-10-16 12:49:46,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629975_1099518554605] hdfs.DFSClient: Slow ReadProcessor read fields took 897ms (threshold=20ms); ack: seqno: 7529 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1724245, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:46,464 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#22 sleep 895 ms because current throughput is 95.24 MB/sec, max allowed is 10.00 MB/sec, already slept 45 time(s) and total slept time is 54484 ms till now. 2015-10-16 12:49:46,660 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629972_1099518554602] hdfs.DFSClient: Slow ReadProcessor read fields took 869ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1568680, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:49:47,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629975_1099518554605] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 7696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1593313, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:47,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 9427ms (threshold=20ms); ack: seqno: 34635 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 660003, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:47,570 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 34646 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 613572, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:47,626 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 34649 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 639191, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:47,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629972_1099518554602] hdfs.DFSClient: Slow ReadProcessor read fields took 899ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1392883, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:49:47,681 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 34650 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 658399, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:47,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 167ms (threshold=20ms); ack: seqno: 34666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790406, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:48,015 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 165ms (threshold=20ms); ack: seqno: 34669 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801681, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:48,345 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 329ms (threshold=20ms); ack: seqno: 34671 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 893142, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:48,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629975_1099518554605] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 7863 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1466472, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:48,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 165ms (threshold=20ms); ack: seqno: 34674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 648299, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:48,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629972_1099518554602] hdfs.DFSClient: Slow ReadProcessor read fields took 894ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1513146, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:49:48,677 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 165ms (threshold=20ms); ack: seqno: 34676 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 644306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:48,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 34680 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 474191, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:48,843 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 34681 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841192, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:49,009 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 164ms (threshold=20ms); ack: seqno: 34685 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 761753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:49,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 164ms (threshold=20ms); ack: seqno: 34689 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 704688, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:49,338 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 164ms (threshold=20ms); ack: seqno: 34694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 858552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:49,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629975_1099518554605] hdfs.DFSClient: Slow ReadProcessor read fields took 896ms (threshold=20ms); ack: seqno: 8030 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1406295, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:49,506 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 166ms (threshold=20ms); ack: seqno: 34705 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 852272, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:49,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629977_1099518554607] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1731175, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:49,670 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 163ms (threshold=20ms); ack: seqno: 34710 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 521735, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:49,748 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629977_1099518554607] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 2275 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 761008, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:49,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 165ms (threshold=20ms); ack: seqno: 34715 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 727967, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:50,000 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 163ms (threshold=20ms); ack: seqno: 34717 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839771, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:50,165 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 163ms (threshold=20ms); ack: seqno: 34724 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 878817, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:50,333 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 160ms (threshold=20ms); ack: seqno: 34745 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 891186, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:50,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629975_1099518554605] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 8199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1602669, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:50,497 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 164ms (threshold=20ms); ack: seqno: 34750 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 696019, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:50,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629977_1099518554607] hdfs.DFSClient: Slow ReadProcessor read fields took 871ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1487091, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:50,661 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 163ms (threshold=20ms); ack: seqno: 34753 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 560130, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:50,984 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 322ms (threshold=20ms); ack: seqno: 34755 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 826138, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:51,149 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 164ms (threshold=20ms); ack: seqno: 34758 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833735, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:51,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 168ms (threshold=20ms); ack: seqno: 34768 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 748822, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:51,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629978_1099518554608] hdfs.DFSClient: Slow ReadProcessor read fields took 896ms (threshold=20ms); ack: seqno: 8369 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1534086, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:51,478 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 159ms (threshold=20ms); ack: seqno: 34771 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 627744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:51,642 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 161ms (threshold=20ms); ack: seqno: 34786 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 872905, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:51,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629977_1099518554607] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1122681, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:51,757 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 34805 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 704267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:51,815 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 34808 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 781966, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:51,874 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 34821 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 556739, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:51,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 34836 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 879444, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:51,988 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 34843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845015, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:52,045 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 34857 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 645246, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:52,162 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 34902 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 659311, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:52,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 34903 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758748, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:52,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629978_1099518554608] hdfs.DFSClient: Slow ReadProcessor read fields took 899ms (threshold=20ms); ack: seqno: 8535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1162281, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:52,540 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629971_1099518554601] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 36611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 515842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:49:52,540 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:49:52,540 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:49:52,540 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:49:52,540 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:49:52,540 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:49:52,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629977_1099518554607] hdfs.DFSClient: Slow ReadProcessor read fields took 899ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1086267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:52,993 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:49:53,032 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024962724 with entries=77930, filesize=122.16 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024992993 2015-10-16 12:49:53,034 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024947989 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024947989 2015-10-16 12:49:53,035 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629979_1099518554609] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992895, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:53,040 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024955410 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024955410 2015-10-16 12:49:53,386 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629979_1099518554609] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 1961 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 660670, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:53,386 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:49:53,386 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:49:53,387 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:49:53,387 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629978_1099518554608] hdfs.DFSClient: Slow ReadProcessor read fields took 902ms (threshold=20ms); ack: seqno: 8702 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1123588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:53,610 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629979_1099518554609] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 2616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874529, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:53,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629977_1099518554607] hdfs.DFSClient: Slow ReadProcessor read fields took 883ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965530, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:54,238 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629979_1099518554609] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 6101 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 412776, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:54,239 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:54,239 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:54,239 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:54,239 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:54,239 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:54,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629978_1099518554608] hdfs.DFSClient: Slow ReadProcessor read fields took 848ms (threshold=20ms); ack: seqno: 8871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1171899, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:54,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629977_1099518554607] hdfs.DFSClient: Slow ReadProcessor read fields took 880ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1164435, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:54,986 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629979_1099518554609] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 10271 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 520411, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:54,986 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:54,986 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:54,986 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:54,986 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:54,986 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:55,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629978_1099518554608] hdfs.DFSClient: Slow ReadProcessor read fields took 869ms (threshold=20ms); ack: seqno: 9038 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1168853, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:55,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629977_1099518554607] hdfs.DFSClient: Slow ReadProcessor read fields took 880ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1140922, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:55,772 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629977_1099518554607] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 3287 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797157, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:55,772 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629979_1099518554609] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 14641 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 436539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:55,772 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:55,772 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:49:55,772 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:49:55,772 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:49:56,365 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629978_1099518554608] hdfs.DFSClient: Slow ReadProcessor read fields took 876ms (threshold=20ms); ack: seqno: 9205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1165588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:56,438 DEBUG [PriorityRpcServer.handler=2,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:49:56,439 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134263920 > lower bound=16777216 2015-10-16 12:49:56,439 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:49:56,467 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629979_1099518554609] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 18508 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 459087, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:56,586 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629979_1099518554609] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 18979 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 561256, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:56,586 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629980_1099518554610] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 93 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 821834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:56,586 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:56,586 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:56,586 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:56,586 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:56,586 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:49:56,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629977_1099518554607] hdfs.DFSClient: Slow ReadProcessor read fields took 848ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1215054, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:57,199 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629979_1099518554609] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 22120 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 539508, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:57,199 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629980_1099518554610] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 990 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819020, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:49:57,200 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:49:57,200 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:49:57,315 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=4799991, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/bea212be6e8846de8166c2996205eb0f 2015-10-16 12:49:57,321 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/bea212be6e8846de8166c2996205eb0f as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bea212be6e8846de8166c2996205eb0f 2015-10-16 12:49:57,326 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bea212be6e8846de8166c2996205eb0f, entries=466270, sequenceid=4799991, filesize=70.8 M 2015-10-16 12:49:57,328 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134282800, currentsize=24.37 MB/25553280 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 889ms, sequenceid=4799991, compaction requested=true 2015-10-16 12:49:57,328 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:3), split_queue=0, merge_queue=0 2015-10-16 12:49:57,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629978_1099518554608] hdfs.DFSClient: Slow ReadProcessor read fields took 875ms (threshold=20ms); ack: seqno: 9372 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1313457, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:57,660 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629977_1099518554607] hdfs.DFSClient: Slow ReadProcessor read fields took 879ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3028656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:57,878 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629979_1099518554609] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 25989 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 549458, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:57,878 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:49:57,878 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:49:57,878 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:49:57,878 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:49:58,379 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629978_1099518554608] hdfs.DFSClient: Slow ReadProcessor read fields took 886ms (threshold=20ms); ack: seqno: 9539 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 18192628, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:58,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629979_1099518554609] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 28577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 761571, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:58,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629977_1099518554607] hdfs.DFSClient: Slow ReadProcessor read fields took 884ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1233482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:58,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629979_1099518554609] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 30386 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 466331, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:58,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629977_1099518554607] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 3799 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827581, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:58,781 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:58,781 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:58,781 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:58,781 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:58,781 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:49:59,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629978_1099518554608] hdfs.DFSClient: Slow ReadProcessor read fields took 893ms (threshold=20ms); ack: seqno: 9706 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1140611, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:49:59,562 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629979_1099518554609] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 35058 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 467015, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:49:59,562 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:49:59,562 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:49:59,562 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:49:59,562 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:49:59,562 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:49:59,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629977_1099518554607] hdfs.DFSClient: Slow ReadProcessor read fields took 840ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1168816, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:00,206 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:50:00,248 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024992993 with entries=77879, filesize=122.09 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025000206 2015-10-16 12:50:00,251 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024962724 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024962724 2015-10-16 12:50:00,252 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629981_1099518554611] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1146981, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:00,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629978_1099518554608] hdfs.DFSClient: Slow ReadProcessor read fields took 882ms (threshold=20ms); ack: seqno: 9873 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1155966, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:00,415 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629981_1099518554611] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 736 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 480329, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:00,415 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:00,415 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:00,415 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629978_1099518554608] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 9897 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729201, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:00,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629977_1099518554607] hdfs.DFSClient: Slow ReadProcessor read fields took 873ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:00,800 DEBUG [PriorityRpcServer.handler=9,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:50:00,800 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:50:00,800 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:4), split_queue=0, merge_queue=0 2015-10-16 12:50:01,196 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629981_1099518554611] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 5013 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 553051, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:01,196 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:01,196 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:01,196 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:01,196 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:01,196 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:01,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629978_1099518554608] hdfs.DFSClient: Slow ReadProcessor read fields took 851ms (threshold=20ms); ack: seqno: 10040 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1092069, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:01,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 865ms (threshold=20ms); ack: seqno: 4181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1708732, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:01,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629981_1099518554611] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 9234 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 477042, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:01,963 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:01,963 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:01,963 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:02,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629978_1099518554608] hdfs.DFSClient: Slow ReadProcessor read fields took 880ms (threshold=20ms); ack: seqno: 10207 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1209165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:02,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 871ms (threshold=20ms); ack: seqno: 4348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1513524, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:02,758 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629981_1099518554611] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 13730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 394833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:02,758 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 4432 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 856081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:02,758 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:02,758 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:02,758 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:03,005 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629981_1099518554611] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 14567 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 660875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:03,370 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629978_1099518554608] hdfs.DFSClient: Slow ReadProcessor read fields took 876ms (threshold=20ms); ack: seqno: 10374 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 9158491, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:03,683 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:03,683 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:03,683 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:03,683 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629981_1099518554611] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 18300 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 431875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:03,683 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:03,683 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:03,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 861ms (threshold=20ms); ack: seqno: 4517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1281468, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:04,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629983_1099518554613] hdfs.DFSClient: Slow ReadProcessor read fields took 873ms (threshold=20ms); ack: seqno: 10543 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1334308, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:04,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629981_1099518554611] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 22833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 424239, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:04,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629983_1099518554613] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 10623 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794247, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:04,461 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:04,461 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:04,461 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:04,461 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:04,461 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:04,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 870ms (threshold=20ms); ack: seqno: 4684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1357564, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:05,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629981_1099518554611] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 27322 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 475068, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:05,291 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:05,291 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:05,291 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:05,291 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:05,291 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:05,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629983_1099518554613] hdfs.DFSClient: Slow ReadProcessor read fields took 835ms (threshold=20ms); ack: seqno: 10710 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1344144, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:05,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 873ms (threshold=20ms); ack: seqno: 4851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1267678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:06,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629981_1099518554611] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 31462 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 606640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:06,028 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:50:06,028 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:50:06,028 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:50:06,028 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:50:06,028 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:50:06,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629983_1099518554613] hdfs.DFSClient: Slow ReadProcessor read fields took 864ms (threshold=20ms); ack: seqno: 10879 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1378078, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:06,690 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 874ms (threshold=20ms); ack: seqno: 5018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 6978471, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:06,817 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 5147 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866601, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:06,817 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629981_1099518554611] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 35869 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 465083, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:06,817 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:06,817 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:06,817 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:06,818 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:07,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629983_1099518554613] hdfs.DFSClient: Slow ReadProcessor read fields took 871ms (threshold=20ms); ack: seqno: 11046 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1383214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:07,386 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629981_1099518554611] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 38563 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 768803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:07,504 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:50:07,548 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025000206 with entries=77897, filesize=122.12 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025007504 2015-10-16 12:50:07,551 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629984_1099518554614] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017934, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:07,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 841ms (threshold=20ms); ack: seqno: 5185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1265419, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:07,759 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:50:07,759 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 5226 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7042095, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:07,760 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629984_1099518554614] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 926 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 678546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:07,760 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:07,760 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:08,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629983_1099518554613] hdfs.DFSClient: Slow ReadProcessor read fields took 877ms (threshold=20ms); ack: seqno: 11215 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1369300, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:08,568 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629983_1099518554613] hdfs.DFSClient: Slow ReadProcessor read fields took 159ms (threshold=20ms); ack: seqno: 11278 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 148589988, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:08,568 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629984_1099518554614] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 5498 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 418377, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:08,568 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:50:08,568 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:50:08,568 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:50:08,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 837ms (threshold=20ms); ack: seqno: 5352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1071461, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:09,365 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629983_1099518554613] hdfs.DFSClient: Slow ReadProcessor read fields took 776ms (threshold=20ms); ack: seqno: 11382 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1324542, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:09,440 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629983_1099518554613] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 11429 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 843669, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:09,440 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629984_1099518554614] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 10082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 470486, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:09,440 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:50:09,440 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:50:09,440 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:50:09,440 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:50:09,441 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:50:09,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 875ms (threshold=20ms); ack: seqno: 5521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1289106, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:10,299 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629984_1099518554614] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 14543 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 533099, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:10,299 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:50:10,299 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:50:10,299 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:50:10,299 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:50:10,364 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629983_1099518554613] hdfs.DFSClient: Slow ReadProcessor read fields took 835ms (threshold=20ms); ack: seqno: 11549 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1314438, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:10,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 874ms (threshold=20ms); ack: seqno: 5688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1208922, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:11,077 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629984_1099518554614] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 18660 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 602728, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:11,077 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:50:11,077 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:50:11,077 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:50:11,077 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:50:11,077 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:50:11,364 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629983_1099518554613] hdfs.DFSClient: Slow ReadProcessor read fields took 872ms (threshold=20ms); ack: seqno: 11716 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1488354, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:11,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 884ms (threshold=20ms); ack: seqno: 5855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 14662838, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:11,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629984_1099518554614] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 22083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701581, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:11,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629984_1099518554614] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 22833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 642346, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:11,996 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 62ms (threshold=20ms) 2015-10-16 12:50:11,996 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 62ms (threshold=20ms) 2015-10-16 12:50:11,996 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 62ms (threshold=20ms) 2015-10-16 12:50:11,996 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 12:50:11,996 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 62ms (threshold=20ms) 2015-10-16 12:50:12,364 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629983_1099518554613] hdfs.DFSClient: Slow ReadProcessor read fields took 868ms (threshold=20ms); ack: seqno: 11883 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1467060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:12,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 896ms (threshold=20ms); ack: seqno: 6022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1192788, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:12,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 6179 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2617956, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:12,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629984_1099518554614] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 27090 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 666407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:12,835 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:12,835 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:12,835 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:12,835 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:12,835 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:13,364 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629983_1099518554613] hdfs.DFSClient: Slow ReadProcessor read fields took 870ms (threshold=20ms); ack: seqno: 12050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1443044, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:13,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 845ms (threshold=20ms); ack: seqno: 6188 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1313857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:13,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629984_1099518554614] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 31818 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 559307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:13,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629982_1099518554612] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 6219 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 744699, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:13,746 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:13,746 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:13,746 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:13,746 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:14,365 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629983_1099518554613] hdfs.DFSClient: Slow ReadProcessor read fields took 878ms (threshold=20ms); ack: seqno: 12217 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1523233, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:14,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629985_1099518554615] hdfs.DFSClient: Slow ReadProcessor read fields took 847ms (threshold=20ms); ack: seqno: 6358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1577657, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:50:15,365 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629983_1099518554613] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 12384 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1731064, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:15,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629985_1099518554615] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 6525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1758908, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:50:15,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629985_1099518554615] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 6643 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762429, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:50:16,364 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629986_1099518554616] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 12553 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1627816, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:16,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629985_1099518554615] hdfs.DFSClient: Slow ReadProcessor read fields took 860ms (threshold=20ms); ack: seqno: 6694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1514766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:50:17,374 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629986_1099518554616] hdfs.DFSClient: Slow ReadProcessor read fields took 906ms (threshold=20ms); ack: seqno: 12720 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 11128152, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:17,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629985_1099518554615] hdfs.DFSClient: Slow ReadProcessor read fields took 896ms (threshold=20ms); ack: seqno: 6861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1278398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:50:17,718 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#24 average throughput is 9.54 MB/sec, slept 41 time(s) and total slept time is 38111 ms. 1 active compactions remaining, total limit is 20.00 MB/sec 2015-10-16 12:50:17,738 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/00c4e1876ef545f8b1781f7a7f2edcf2 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/00c4e1876ef545f8b1781f7a7f2edcf2 2015-10-16 12:50:17,753 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629984_1099518554614] hdfs.DFSClient: Slow ReadProcessor read fields took 3596ms (threshold=20ms); ack: seqno: 34142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859042, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:17,753 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:50:17,754 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:50:17,764 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1dff2ad0156b4f9d892326753ed03bec, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1dff2ad0156b4f9d892326753ed03bec 2015-10-16 12:50:17,772 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/87bb3be33dc644ca8b7053ae20aeb63e, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/87bb3be33dc644ca8b7053ae20aeb63e 2015-10-16 12:50:17,780 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a5a9ed6be16e40758be730a06fdc5592, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a5a9ed6be16e40758be730a06fdc5592 2015-10-16 12:50:17,789 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/740dc9f44e214ffd83d6ca72fd54f082, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/740dc9f44e214ffd83d6ca72fd54f082 2015-10-16 12:50:17,789 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 00c4e1876ef545f8b1781f7a7f2edcf2(size=425.0 M), total size for store is 7.1 G. This selection was in queue for 0sec, and took 43sec to execute. 2015-10-16 12:50:17,789 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=425.0 M, priority=2, time=36902883215143964; duration=43sec 2015-10-16 12:50:17,789 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:4), split_queue=0, merge_queue=0 2015-10-16 12:50:17,789 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 4 compacting, 3 eligible, 10 blocking 2015-10-16 12:50:17,791 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:50:17,791 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:50:17,791 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:50:17,791 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 4 compacting, 3 eligible, 10 blocking 2015-10-16 12:50:17,793 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:50:17,793 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:50:17,793 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:50:17,793 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 4 compacting, 3 eligible, 10 blocking 2015-10-16 12:50:17,795 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:50:17,795 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:50:17,795 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:50:17,795 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 4 compacting, 3 eligible, 10 blocking 2015-10-16 12:50:17,797 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:50:17,797 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:50:17,797 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:50:17,902 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536915936 > lower bound=16777216 2015-10-16 12:50:17,902 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=512.04 MB 2015-10-16 12:50:18,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629984_1099518554614] hdfs.DFSClient: Slow ReadProcessor read fields took 248ms (threshold=20ms); ack: seqno: 34143 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 852332, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:18,364 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629986_1099518554616] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 12887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1385241, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:18,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629987_1099518554617] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 1204 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 657390, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:50:18,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629986_1099518554616] hdfs.DFSClient: Slow ReadProcessor read fields took 395ms (threshold=20ms); ack: seqno: 13054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1406067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:19,364 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629986_1099518554616] hdfs.DFSClient: Slow ReadProcessor read fields took 389ms (threshold=20ms); ack: seqno: 13221 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1314788, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:19,865 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629986_1099518554616] hdfs.DFSClient: Slow ReadProcessor read fields took 361ms (threshold=20ms); ack: seqno: 13390 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1246409, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:20,365 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629986_1099518554616] hdfs.DFSClient: Slow ReadProcessor read fields took 349ms (threshold=20ms); ack: seqno: 13559 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1376376, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:20,714 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=4986430, memsize=512.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/f428680ab2dc4a3b8d137ce10d235ce6 2015-10-16 12:50:20,720 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/f428680ab2dc4a3b8d137ce10d235ce6 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f428680ab2dc4a3b8d137ce10d235ce6 2015-10-16 12:50:20,734 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f428680ab2dc4a3b8d137ce10d235ce6, entries=1864340, sequenceid=4986430, filesize=283.2 M 2015-10-16 12:50:20,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629984_1099518554614] hdfs.DFSClient: Slow ReadProcessor read fields took 2734ms (threshold=20ms); ack: seqno: 34144 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 771929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:20,736 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~512.04 MB/536915520, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2834ms, sequenceid=4986430, compaction requested=true 2015-10-16 12:50:20,736 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:50:20,736 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 4 compacting, 4 eligible, 10 blocking 2015-10-16 12:50:20,738 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 1113905069 starting at candidate #0 after considering 3 permutations with 2 in ratio 2015-10-16 12:50:20,738 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:50:20,738 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:50:20,738 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=1.0 G 2015-10-16 12:50:20,738 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/00c4e1876ef545f8b1781f7a7f2edcf2, keycount=2797620, bloomtype=ROW, size=425.0 M, encoding=NONE, seqNum=4566918 2015-10-16 12:50:20,738 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d5cb1dd6386b41e3a42ea2835054ad88, keycount=1864370, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=4753360 2015-10-16 12:50:20,738 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bea212be6e8846de8166c2996205eb0f, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=4799991 2015-10-16 12:50:20,739 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f428680ab2dc4a3b8d137ce10d235ce6, keycount=1864340, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=4986430 2015-10-16 12:50:20,759 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:50:20,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629986_1099518554616] hdfs.DFSClient: Slow ReadProcessor read fields took 373ms (threshold=20ms); ack: seqno: 13726 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1495719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:21,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1172732, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:21,869 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 723305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:21,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629986_1099518554616] hdfs.DFSClient: Slow ReadProcessor read fields took 893ms (threshold=20ms); ack: seqno: 13893 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1395142, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:22,281 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629984_1099518554614] hdfs.DFSClient: Slow ReadProcessor read fields took 1545ms (threshold=20ms); ack: seqno: 34145 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:22,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 875ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1210022, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:22,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629986_1099518554616] hdfs.DFSClient: Slow ReadProcessor read fields took 879ms (threshold=20ms); ack: seqno: 14060 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047675, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:22,987 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629984_1099518554614] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 37887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 574525, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:22,987 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629986_1099518554616] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 14142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838675, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:22,987 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:22,987 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:22,987 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:22,987 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:22,988 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:23,210 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:50:23,247 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025007504 with entries=77883, filesize=122.09 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025023211 2015-10-16 12:50:23,248 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024992993 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445024992993 2015-10-16 12:50:23,255 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025000206 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025000206 2015-10-16 12:50:23,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 883ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1413516, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:23,869 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629991_1099518554621] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 3347 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 529861, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:23,869 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 604 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866330, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:23,869 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:23,869 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:23,869 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:23,869 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:23,869 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:23,871 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629986_1099518554616] hdfs.DFSClient: Slow ReadProcessor read fields took 825ms (threshold=20ms); ack: seqno: 14227 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196593, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:24,204 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629991_1099518554621] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 4456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741556, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:24,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 847ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1222092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:24,833 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629991_1099518554621] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 7748 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 571080, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:24,833 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 709 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 754754, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:24,833 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:24,833 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:24,833 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:24,833 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:24,833 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:24,869 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629991_1099518554621] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 7767 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 32991707, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:24,869 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:24,869 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:24,869 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:24,869 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:24,869 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:24,871 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629986_1099518554616] hdfs.DFSClient: Slow ReadProcessor read fields took 886ms (threshold=20ms); ack: seqno: 14394 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1104895, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:25,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629991_1099518554621] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 12131 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 496095, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:25,689 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:25,689 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:25,689 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:25,689 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:25,689 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:25,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 819ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1348220, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:25,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629986_1099518554616] hdfs.DFSClient: Slow ReadProcessor read fields took 855ms (threshold=20ms); ack: seqno: 14561 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1133913, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:26,517 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629991_1099518554621] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 16297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 574656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:26,518 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:26,518 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:26,518 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:26,518 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:26,518 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:26,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 849ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1326573, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:26,799 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 1001 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1880829, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:26,852 DEBUG [PriorityRpcServer.handler=11,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:50:26,852 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134270496 > lower bound=16777216 2015-10-16 12:50:26,852 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:50:26,871 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 868ms (threshold=20ms); ack: seqno: 14730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1249072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:26,878 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629991_1099518554621] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 18203 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 573933, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:27,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629993_1099518554623] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 431 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047421, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:27,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629991_1099518554621] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 19617 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 562804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:27,195 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:27,195 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:27,195 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:27,195 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:27,195 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:27,264 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629991_1099518554621] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 19661 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 723784, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:27,646 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=5033061, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/3ec6ad24fd4d49e48a4cede70b168599 2015-10-16 12:50:27,652 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/3ec6ad24fd4d49e48a4cede70b168599 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3ec6ad24fd4d49e48a4cede70b168599 2015-10-16 12:50:27,658 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3ec6ad24fd4d49e48a4cede70b168599, entries=466270, sequenceid=5033061, filesize=70.8 M 2015-10-16 12:50:27,660 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134281600, currentsize=19.10 MB/20026960 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 808ms, sequenceid=5033061, compaction requested=false 2015-10-16 12:50:27,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 874ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1421193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:27,871 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 861ms (threshold=20ms); ack: seqno: 14897 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1075630, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:27,929 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 14923 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5786071, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:27,929 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629991_1099518554621] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 22947 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5643781, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:27,929 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:27,929 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:27,929 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:27,929 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:27,929 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:28,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629991_1099518554621] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 25184 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 677431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:28,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 869ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1455952, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:28,898 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:28,898 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 1464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 713247, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:28,899 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:28,899 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:28,899 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:28,899 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:28,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 875ms (threshold=20ms); ack: seqno: 15064 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1092858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:29,754 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629991_1099518554621] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 32197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 412114, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:29,754 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:29,754 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:29,754 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:29,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 842ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1574835, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:29,871 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 852ms (threshold=20ms); ack: seqno: 15231 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1253553, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:30,518 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629991_1099518554621] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 36418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 512759, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:30,518 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:30,518 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:30,518 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:30,519 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:30,519 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:30,773 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 859ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7829310, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:30,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 850ms (threshold=20ms); ack: seqno: 15398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101259, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:30,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629991_1099518554621] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 38394 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672980, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:30,996 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:50:31,045 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:50:31,080 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025023211 with entries=78196, filesize=122.58 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025030997 2015-10-16 12:50:31,082 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025007504 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025007504 2015-10-16 12:50:31,083 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629994_1099518554624] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917659, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:31,367 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:31,367 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629994_1099518554624] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 1412 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 411217, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:31,367 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:31,367 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:31,367 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:31,367 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:31,539 DEBUG [PriorityRpcServer.handler=3,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:50:31,539 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134262688 > lower bound=16777216 2015-10-16 12:50:31,539 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:50:31,562 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629994_1099518554624] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 2428 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 505543, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:31,767 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 867ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1135545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:31,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 852ms (threshold=20ms); ack: seqno: 15567 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1104020, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:32,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629995_1099518554625] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 590 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 737334, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:50:32,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 15679 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728754, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:32,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629994_1099518554624] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 4617 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 523188, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:32,003 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:32,003 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:32,003 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:32,003 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:32,003 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:32,371 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=5079687, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d0fe8512faa446e090b42ede97699a43 2015-10-16 12:50:32,378 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d0fe8512faa446e090b42ede97699a43 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d0fe8512faa446e090b42ede97699a43 2015-10-16 12:50:32,378 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629994_1099518554624] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 6007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 670752, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:32,391 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d0fe8512faa446e090b42ede97699a43, entries=466220, sequenceid=5079687, filesize=70.8 M 2015-10-16 12:50:32,392 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134266880, currentsize=20.09 MB/21069440 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 853ms, sequenceid=5079687, compaction requested=true 2015-10-16 12:50:32,393 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:50:32,767 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 807ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1294040, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:32,862 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629994_1099518554624] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 8551 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1177992, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:32,862 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629990_1099518554620] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 2080 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 848980, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:32,862 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:50:32,862 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:50:32,862 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:50:32,862 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:50:32,862 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:50:32,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 806ms (threshold=20ms); ack: seqno: 15734 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1269376, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:33,705 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629994_1099518554624] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 13117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 459091, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:33,705 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:33,705 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:33,705 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:33,705 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:33,705 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:33,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629996_1099518554626] hdfs.DFSClient: Slow ReadProcessor read fields took 796ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972622, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:50:33,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 868ms (threshold=20ms); ack: seqno: 15901 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1093238, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:34,501 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629994_1099518554624] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 17328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 494533, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:34,501 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:34,501 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:34,501 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:34,501 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:34,501 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:34,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629996_1099518554626] hdfs.DFSClient: Slow ReadProcessor read fields took 856ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950275, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:50:34,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 868ms (threshold=20ms); ack: seqno: 16068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1231061, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:35,270 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629994_1099518554624] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 21499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 530984, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:35,270 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:35,270 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:35,270 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:35,270 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:35,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629996_1099518554626] hdfs.DFSClient: Slow ReadProcessor read fields took 856ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013201, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:50:35,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 877ms (threshold=20ms); ack: seqno: 16237 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1275702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:36,071 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629994_1099518554624] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 25140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934941, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:36,202 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629994_1099518554624] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 25707 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 503522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:36,202 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:36,202 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:36,202 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:36,202 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:36,202 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:36,224 DEBUG [PriorityRpcServer.handler=14,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:50:36,224 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:50:36,224 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 12:50:36,767 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629996_1099518554626] hdfs.DFSClient: Slow ReadProcessor read fields took 854ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1065308, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:50:36,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 885ms (threshold=20ms); ack: seqno: 16404 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1245143, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:37,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629994_1099518554624] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 29934 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 609791, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:37,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 16550 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 690905, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:37,007 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:37,007 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:37,007 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:37,007 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:37,007 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:37,767 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629996_1099518554626] hdfs.DFSClient: Slow ReadProcessor read fields took 861ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085205, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:50:37,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 850ms (threshold=20ms); ack: seqno: 16571 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1344072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:37,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629996_1099518554626] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 2990 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 11255442, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:50:37,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629994_1099518554624] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 34720 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 625117, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:37,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629992_1099518554622] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 16580 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967647, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:37,913 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:37,913 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:37,913 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:37,913 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:37,913 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:38,636 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:50:38,679 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025030997 with entries=77958, filesize=122.21 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025038636 2015-10-16 12:50:38,682 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1015194, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:38,683 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025023211 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025023211 2015-10-16 12:50:38,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 440 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 440500, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:38,795 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:38,795 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:38,795 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:38,795 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:38,795 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:38,797 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629996_1099518554626] hdfs.DFSClient: Slow ReadProcessor read fields took 876ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1422038, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:50:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 12:50:38,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629997_1099518554627] hdfs.DFSClient: Slow ReadProcessor read fields took 843ms (threshold=20ms); ack: seqno: 16740 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1226997, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:50:39,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 2518 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 568835, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:39,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629997_1099518554627] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 17081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 875595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:50:39,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629996_1099518554626] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 3526 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863174, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:50:39,213 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:39,213 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:39,213 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:39,213 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:39,213 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:39,610 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 4454 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 622802, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:39,610 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629996_1099518554626] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 3960 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:50:39,610 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629997_1099518554627] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 17526 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797646, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:50:39,610 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:39,610 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:39,610 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:39,610 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:39,610 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:39,996 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 6321 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 702808, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:39,996 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629999_1099518554629] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 4393 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 890943, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:50:39,996 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:39,996 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:39,996 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629997_1099518554627] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 17989 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 721746, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:50:39,996 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:39,996 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:39,996 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:50:40,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629999_1099518554629] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 4820 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3599158, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:50:40,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 8195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 383105, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:40,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629997_1099518554627] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18449 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2492528, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:50:40,372 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:40,372 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:40,372 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:40,372 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:40,653 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 9064 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 724280, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:40,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630000_1099518554630] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 18972 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 755908, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:50:40,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629999_1099518554629] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 5332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 840735, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:50:40,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 9515 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 545640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:40,778 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:40,778 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:40,779 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:40,779 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:41,174 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630000_1099518554630] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 19444 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 678843, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:50:41,174 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 11172 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 506567, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:41,174 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629999_1099518554629] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 5806 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 730509, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:50:41,174 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:41,174 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:41,174 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:41,174 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:41,174 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:41,570 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630001_1099518554631] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 6264 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2345789, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:41,570 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630000_1099518554630] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 19907 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787434, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:50:41,570 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 12909 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 625868, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:41,570 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:41,570 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:41,570 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:41,570 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:41,570 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:41,965 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630001_1099518554631] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 6736 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 739130, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:41,965 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630000_1099518554630] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 20396 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 704424, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:50:41,966 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 14580 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 550816, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:41,966 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:42,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630001_1099518554631] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 7203 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812515, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:42,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630002_1099518554632] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 20856 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 6445612, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:42,359 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 16290 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 757339, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:42,359 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:42,359 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:42,359 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:42,359 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:42,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630002_1099518554632] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 21331 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 746089, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:42,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630001_1099518554631] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 7681 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 731714, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:42,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 18001 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728151, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:42,742 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:42,742 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:42,743 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:42,743 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:42,743 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:43,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630001_1099518554631] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 7996 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 846941, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:43,121 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630002_1099518554632] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 21804 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 848297, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:43,121 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630001_1099518554631] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 8139 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887760, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:43,121 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 19733 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 695865, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:43,121 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:50:43,121 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:50:43,121 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:50:43,121 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:43,121 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:50:43,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630002_1099518554632] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 22266 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865880, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:43,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 21425 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 561799, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:43,512 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:50:43,513 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630003_1099518554633] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 8611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885300, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:43,897 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630002_1099518554632] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 22740 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 704959, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:43,897 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630003_1099518554633] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 9102 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965494, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:43,897 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 23080 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 690919, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:43,897 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:50:43,897 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:50:43,897 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:50:43,898 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:50:43,898 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:50:44,278 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630003_1099518554633] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 9576 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 840212, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:44,278 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 24804 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 693579, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:44,278 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630004_1099518554634] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 23198 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 763081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:44,278 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:44,278 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:44,278 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:44,278 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:44,278 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:44,661 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630003_1099518554633] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 10051 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 850946, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:50:44,661 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 26538 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 612278, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:44,661 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630004_1099518554634] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 23640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 739337, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:44,661 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:44,661 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:44,661 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:44,661 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:50:44,661 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:45,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630005_1099518554635] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 10527 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 759077, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:50:45,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 28280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842204, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:45,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630004_1099518554634] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 24101 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868781, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:45,048 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:45,048 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:45,048 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:45,048 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:45,048 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:45,439 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 30146 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 509815, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:45,439 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630005_1099518554635] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 10982 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 763549, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:50:45,439 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630004_1099518554634] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 24551 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 800985, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:50:45,439 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:45,439 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:45,439 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:45,439 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:45,439 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:50:45,856 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:45,856 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:45,856 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:45,856 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:50:46,195 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:50:46,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 33339 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 617773, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:46,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630005_1099518554635] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 11771 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869303, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:50:46,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630006_1099518554636] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 25353 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841657, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:46,195 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:50:46,195 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:50:46,195 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:50:46,195 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:50:46,320 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630006_1099518554636] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 25477 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787251, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:46,320 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 33797 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 516614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:46,320 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:46,320 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630005_1099518554635] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 11891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762767, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:50:46,320 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:46,320 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:46,320 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:50:46,359 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#22 9586886/6992600 (137.10%), rate=18345.83 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=unlimited, activeCompactions=2] 2015-10-16 12:50:46,701 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630005_1099518554635] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 12340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947304, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:50:46,701 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 35565 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 872168, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:46,701 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630006_1099518554636] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 25946 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902353, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:46,702 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:46,702 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:46,702 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:46,702 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:46,702 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:47,104 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630006_1099518554636] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 26488 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1057907, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:47,104 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630007_1099518554637] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 12850 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898427, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:47,106 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 36785 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 889098, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:47,475 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630007_1099518554637] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 13301 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857260, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:47,475 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080629998_1099518554628] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 38510 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 514676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:47,475 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630006_1099518554636] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 26973 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 826176, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:50:47,475 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:50:47,475 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:50:47,476 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:50:47,476 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:50:47,476 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:50:47,554 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:50:47,604 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025038636 with entries=77893, filesize=122.11 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025047555 2015-10-16 12:50:47,607 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 742175, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:47,878 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:50:47,878 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:50:47,878 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:50:47,880 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630007_1099518554637] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 13741 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909850, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:47,880 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630009_1099518554639] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 27446 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1015779, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:50:48,250 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 3066 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:48,250 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630009_1099518554639] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 27903 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 872980, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:50:48,250 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630007_1099518554637] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 14179 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 724350, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:50:48,250 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:50:48,250 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:50:48,250 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:50:48,250 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:50:48,250 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:50:48,629 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630010_1099518554640] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 14620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4091271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:50:48,629 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630009_1099518554639] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 28368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:50:48,629 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 4881 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 673649, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:48,630 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:50:48,630 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:50:48,630 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:50:48,631 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:50:49,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 6610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 723772, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:49,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630009_1099518554639] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 28865 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 680279, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:50:49,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630010_1099518554640] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 15056 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974042, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:50:49,019 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:49,019 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:49,019 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:49,019 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:49,019 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:49,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 8386 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 605098, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:49,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630010_1099518554640] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 15491 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 891519, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:50:49,405 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:50:49,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630011_1099518554641] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 29305 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2672344, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:49,405 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:50:49,405 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:49,405 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:50:49,405 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:49,793 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630011_1099518554641] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 29860 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 815263, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:49,793 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 10141 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 710708, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:49,793 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630010_1099518554640] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 15911 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1182665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:50:49,793 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:49,793 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:49,793 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:49,793 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:49,793 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:49,868 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 10151 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 686684, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:49,868 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 12:50:49,868 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 12:50:49,868 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 12:50:49,868 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 12:50:49,868 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 12:50:50,213 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630010_1099518554640] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 16400 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:50:50,213 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630011_1099518554641] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 30362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:50,213 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 11708 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 547513, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:50,213 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:50,213 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:50:50,213 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:50,213 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:50,213 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:50,508 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#22 average throughput is 14.71 MB/sec, slept 100 time(s) and total slept time is 99702 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 12:50:50,762 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 14572 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 449616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:50,762 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:50,762 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:50,762 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:50:51,251 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630011_1099518554641] hdfs.DFSClient: Slow ReadProcessor read fields took 739ms (threshold=20ms); ack: seqno: 30675 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 953839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:50:51,251 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hdfs.DFSClient: Slow waitForAckedSeqno took 740ms (threshold=20ms) 2015-10-16 12:50:51,269 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/eb253d4644d442638f3ed2235176bedd as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eb253d4644d442638f3ed2235176bedd 2015-10-16 12:50:51,285 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:50:51,286 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:50:51,295 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fb8e84ddaef844f78ab1023bd41241e0, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fb8e84ddaef844f78ab1023bd41241e0 2015-10-16 12:50:51,303 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a7ab7d2869bf43c2b55cd843cdea0421, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a7ab7d2869bf43c2b55cd843cdea0421 2015-10-16 12:50:51,311 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a11f78abbe6848f09c630fb9e406e9f2, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a11f78abbe6848f09c630fb9e406e9f2 2015-10-16 12:50:51,320 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a1cbf15689154134b82f6dc5cd6572c6, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a1cbf15689154134b82f6dc5cd6572c6 2015-10-16 12:50:51,320 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into eb253d4644d442638f3ed2235176bedd(size=1.8 G), total size for store is 7.5 G. This selection was in queue for 0sec, and took 2mins, 5sec to execute. 2015-10-16 12:50:51,320 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=1.8 G, priority=5, time=36902834780294351; duration=2mins, 5sec 2015-10-16 12:50:51,320 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 12:50:51,320 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 4 compacting, 2 eligible, 10 blocking 2015-10-16 12:50:51,321 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:50:51,321 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:50:51,321 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:50:51,321 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 4 compacting, 2 eligible, 10 blocking 2015-10-16 12:50:51,322 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:50:51,322 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:50:51,322 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:50:51,630 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630012_1099518554642] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 17274 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 658276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:50:51,630 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536928336 > lower bound=16777216 2015-10-16 12:50:51,630 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 12:50:51,638 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#25 average throughput is 33.45 MB/sec, slept 18 time(s) and total slept time is 15354 ms. 0 active compactions remaining, total limit is unlimited 2015-10-16 12:50:51,651 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1b495f04e8814fdfb38eec4b2baacfec as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1b495f04e8814fdfb38eec4b2baacfec 2015-10-16 12:50:51,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 312ms (threshold=20ms); ack: seqno: 17913 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701919, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:51,658 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:50:51,663 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:50:51,670 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/00c4e1876ef545f8b1781f7a7f2edcf2, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/00c4e1876ef545f8b1781f7a7f2edcf2 2015-10-16 12:50:51,678 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d5cb1dd6386b41e3a42ea2835054ad88, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d5cb1dd6386b41e3a42ea2835054ad88 2015-10-16 12:50:51,686 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bea212be6e8846de8166c2996205eb0f, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bea212be6e8846de8166c2996205eb0f 2015-10-16 12:50:51,695 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f428680ab2dc4a3b8d137ce10d235ce6, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f428680ab2dc4a3b8d137ce10d235ce6 2015-10-16 12:50:51,695 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 1b495f04e8814fdfb38eec4b2baacfec(size=1.0 G), total size for store is 7.5 G. This selection was in queue for 0sec, and took 30sec to execute. 2015-10-16 12:50:51,695 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=1.0 G, priority=2, time=36902929592204944; duration=30sec 2015-10-16 12:50:51,695 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:50:54,536 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=5266131, memsize=512.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c7e1590f6a494cf48a1a0b7ff48e8e97 2015-10-16 12:50:54,542 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c7e1590f6a494cf48a1a0b7ff48e8e97 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c7e1590f6a494cf48a1a0b7ff48e8e97 2015-10-16 12:50:54,548 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c7e1590f6a494cf48a1a0b7ff48e8e97, entries=1864390, sequenceid=5266131, filesize=283.2 M 2015-10-16 12:50:54,549 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 2891ms (threshold=20ms); ack: seqno: 17915 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 665459, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:54,550 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536927920, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2919ms, sequenceid=5266131, compaction requested=true 2015-10-16 12:50:54,550 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:50:54,550 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 6 store files, 0 compacting, 6 eligible, 10 blocking 2015-10-16 12:50:54,553 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 10 permutations with 0 in ratio 2015-10-16 12:50:54,553 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:50:54,553 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:50:55,473 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 924ms (threshold=20ms); ack: seqno: 17916 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941670, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:56,156 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 21288 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975686, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:56,764 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 24248 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 805059, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:57,050 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 25596 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 614770, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:57,050 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:57,050 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:57,051 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:57,050 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:50:57,822 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 29792 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825710, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:58,088 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 208ms (threshold=20ms); ack: seqno: 30135 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 208429225, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:58,088 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 209ms (threshold=20ms) 2015-10-16 12:50:58,088 INFO [sync.2] wal.FSHLog: Slow sync cost: 208 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:58,088 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 209ms (threshold=20ms) 2015-10-16 12:50:58,088 INFO [sync.4] wal.FSHLog: Slow sync cost: 208 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:58,088 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 209ms (threshold=20ms) 2015-10-16 12:50:58,088 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 209ms (threshold=20ms) 2015-10-16 12:50:58,088 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 209ms (threshold=20ms) 2015-10-16 12:50:58,088 INFO [sync.1] wal.FSHLog: Slow sync cost: 209 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:58,088 INFO [sync.3] wal.FSHLog: Slow sync cost: 209 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:58,088 INFO [sync.0] wal.FSHLog: Slow sync cost: 209 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:58,308 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 31170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 412179, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:58,308 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:58,308 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:50:58,308 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:50:58,308 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:50:59,295 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630008_1099518554638] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 36757 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 580296, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:50:59,310 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:50:59,310 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:50:59,310 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:50:59,310 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:50:59,310 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:50:59,711 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:50:59,759 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:50:59,778 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025047555 with entries=78210, filesize=122.60 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025059711 2015-10-16 12:50:59,780 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025030997 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025030997 2015-10-16 12:50:59,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1278683, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:50:59,786 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025038636 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025038636 2015-10-16 12:51:00,109 DEBUG [PriorityRpcServer.handler=6,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:51:00,109 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134262784 > lower bound=16777216 2015-10-16 12:51:00,109 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:51:00,138 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 1920 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 469657, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:00,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 2661 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 498137, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:00,303 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:51:00,303 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:51:00,303 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:51:00,303 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:51:01,116 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630017_1099518554647] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 642 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741189, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:51:01,116 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 7091 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 506480, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:01,116 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:01,116 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:01,116 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:01,116 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:01,116 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:01,486 WARN [MemStoreFlusher.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:51:01,494 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=5312762, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8990e31289164325a8923446ff5ab967 2015-10-16 12:51:01,499 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8990e31289164325a8923446ff5ab967 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8990e31289164325a8923446ff5ab967 2015-10-16 12:51:01,505 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8990e31289164325a8923446ff5ab967, entries=466260, sequenceid=5312762, filesize=70.8 M 2015-10-16 12:51:01,507 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134279360, currentsize=40.97 MB/42962480 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1398ms, sequenceid=5312762, compaction requested=true 2015-10-16 12:51:01,507 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:51:01,507 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 0 compacting, 7 eligible, 10 blocking 2015-10-16 12:51:01,512 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 15 permutations with 0 in ratio 2015-10-16 12:51:01,512 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:51:01,512 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:51:01,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 10135 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 752328, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:02,065 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 11590 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 588295, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:02,066 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:02,066 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:02,066 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:02,066 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:02,066 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:03,055 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 17110 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 523349, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:03,055 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:03,055 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:03,055 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:03,055 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:03,055 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:04,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 22657 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 545986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:04,063 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:51:04,063 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:51:04,064 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:51:04,063 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:51:04,063 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:51:04,474 DEBUG [PriorityRpcServer.handler=13,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:51:04,474 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134276416 > lower bound=16777216 2015-10-16 12:51:04,474 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:51:04,496 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 25096 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 518471, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:04,896 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630018_1099518554648] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 526 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833756, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:04,896 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 27102 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 557720, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:04,896 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:04,896 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:04,896 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:04,896 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:04,896 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:05,343 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630018_1099518554648] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 1152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 24259116, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:05,344 WARN [MemStoreFlusher.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:51:05,369 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=5359393, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/fd6965ce2ece45309681b00bbe5bd53b 2015-10-16 12:51:05,375 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/fd6965ce2ece45309681b00bbe5bd53b as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fd6965ce2ece45309681b00bbe5bd53b 2015-10-16 12:51:05,397 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fd6965ce2ece45309681b00bbe5bd53b, entries=466270, sequenceid=5359393, filesize=70.8 M 2015-10-16 12:51:05,399 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134281760, currentsize=25.65 MB/26892960 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 925ms, sequenceid=5359393, compaction requested=true 2015-10-16 12:51:05,399 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:51:05,399 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 0 compacting, 8 eligible, 10 blocking 2015-10-16 12:51:05,403 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 7 files of size 3683566381 starting at candidate #1 after considering 21 permutations with 2 in ratio 2015-10-16 12:51:05,403 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:51:05,403 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 0 compacting, 8 eligible, 10 blocking 2015-10-16 12:51:05,407 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 7 files of size 3683566381 starting at candidate #1 after considering 21 permutations with 2 in ratio 2015-10-16 12:51:05,407 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:51:05,407 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:51:05,407 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 7 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=3.4 G 2015-10-16 12:51:05,407 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eb253d4644d442638f3ed2235176bedd, keycount=12404350, bloomtype=ROW, size=1.8 G, encoding=NONE, seqNum=4287132 2015-10-16 12:51:05,407 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1b495f04e8814fdfb38eec4b2baacfec, keycount=6992600, bloomtype=ROW, size=1.0 G, encoding=NONE, seqNum=4986430 2015-10-16 12:51:05,407 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3ec6ad24fd4d49e48a4cede70b168599, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=5033061 2015-10-16 12:51:05,407 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d0fe8512faa446e090b42ede97699a43, keycount=466220, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=5079687 2015-10-16 12:51:05,408 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c7e1590f6a494cf48a1a0b7ff48e8e97, keycount=1864390, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=5266131 2015-10-16 12:51:05,408 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8990e31289164325a8923446ff5ab967, keycount=466260, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=5312762 2015-10-16 12:51:05,408 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fd6965ce2ece45309681b00bbe5bd53b, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=5359393 2015-10-16 12:51:05,442 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:51:05,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630019_1099518554649] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 164 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 694366, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:05,609 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 30648 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 661467, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:05,609 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:05,609 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:05,609 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:05,609 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:05,609 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:06,142 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 33484 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 500367, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:06,142 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630019_1099518554649] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 837 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 690193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:06,143 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:06,143 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:06,143 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:06,395 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 34186 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 689300, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:06,726 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 35837 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 472542, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:06,726 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630019_1099518554649] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 1628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 723804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:06,726 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:06,726 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:06,726 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:06,726 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:06,726 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:07,221 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:51:07,250 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630020_1099518554650] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 2304 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 798397, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:07,250 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 38600 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 444453, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:07,250 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:07,250 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:07,250 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:07,250 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:07,250 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:07,765 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 503ms (threshold=20ms) 2015-10-16 12:51:07,769 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Slow sync cost: 507 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:07,777 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630016_1099518554646] hdfs.DFSClient: Slow ReadProcessor read fields took 157ms (threshold=20ms); ack: seqno: 40625 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 21928834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:07,790 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 170ms (threshold=20ms) 2015-10-16 12:51:07,790 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 170ms (threshold=20ms) 2015-10-16 12:51:07,790 INFO [sync.3] wal.FSHLog: Slow sync cost: 170 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:07,790 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 170ms (threshold=20ms) 2015-10-16 12:51:07,790 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 169ms (threshold=20ms) 2015-10-16 12:51:07,790 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 169ms (threshold=20ms) 2015-10-16 12:51:07,790 INFO [sync.4] wal.FSHLog: Slow sync cost: 170 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:07,790 INFO [sync.2] wal.FSHLog: Slow sync cost: 169 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:07,790 INFO [sync.0] wal.FSHLog: Slow sync cost: 169 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:07,803 INFO [sync.1] wal.FSHLog: Slow sync cost: 182 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:07,804 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025059711 with entries=81669, filesize=128.02 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025067221 2015-10-16 12:51:07,805 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025047555 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025047555 2015-10-16 12:51:07,806 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 970682, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:07,854 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:07,854 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:07,855 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 103 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 615973, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:07,855 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630020_1099518554650] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 3100 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 687076, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:07,855 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:08,388 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 2885 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1422905, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:08,388 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630020_1099518554650] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 3776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 742261, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:08,388 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:08,388 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:08,388 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:08,388 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:08,388 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:08,694 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630020_1099518554650] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 4166 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 21343089, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:08,923 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 5626 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 539311, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:08,923 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630023_1099518554653] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 4458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827920, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:08,923 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:08,923 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:08,923 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:08,923 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:08,923 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:09,292 DEBUG [PriorityRpcServer.handler=14,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:51:09,292 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134278464 > lower bound=16777216 2015-10-16 12:51:09,292 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:51:09,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 7712 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 601301, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:09,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 8201 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 517419, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:09,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630024_1099518554654] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 137 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914226, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:09,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630023_1099518554653] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 5100 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1086024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:09,447 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:09,447 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:09,447 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:09,447 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:09,890 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630024_1099518554654] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 748 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844191, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:09,890 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630023_1099518554653] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 5631 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870304, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:09,891 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 10344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 478322, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:09,891 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:09,891 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:09,891 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:09,891 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:09,891 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:10,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=5406026, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5fb2092b29984971b29aa5bceef5a2e9 2015-10-16 12:51:10,166 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5fb2092b29984971b29aa5bceef5a2e9 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5fb2092b29984971b29aa5bceef5a2e9 2015-10-16 12:51:10,172 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5fb2092b29984971b29aa5bceef5a2e9, entries=466290, sequenceid=5406026, filesize=70.8 M 2015-10-16 12:51:10,173 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134287840, currentsize=23.40 MB/24540160 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 881ms, sequenceid=5406026, compaction requested=false 2015-10-16 12:51:10,347 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 12695 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 418953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:10,347 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630023_1099518554653] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 6205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 889962, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:10,347 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:51:10,347 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:51:10,347 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:51:10,347 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:51:10,347 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:51:10,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 15351 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 426178, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:10,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630025_1099518554655] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 6904 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 680127, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:10,858 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:10,858 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:10,858 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:10,858 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:10,858 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:11,189 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 16539 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692460, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:11,414 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:11,414 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 17653 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 511591, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:11,414 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630025_1099518554655] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 7689 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862312, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:11,414 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:11,414 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:11,414 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:11,414 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:11,943 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630026_1099518554656] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 8342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2666299, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:11,943 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 20348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 485815, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:11,944 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:11,944 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:11,944 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:11,944 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:11,944 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:12,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630026_1099518554656] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 9097 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758988, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:12,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 22979 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 495747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:12,457 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:12,457 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:12,457 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:12,457 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:12,457 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:12,993 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630026_1099518554656] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 9793 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 856348, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:12,993 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 25693 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 546883, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:12,994 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:12,994 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:12,994 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:12,994 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:12,994 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:13,521 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 28345 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 445223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:13,521 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630027_1099518554657] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 10507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736621, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:13,521 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:13,521 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:13,521 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:13,521 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:13,521 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:13,997 DEBUG [PriorityRpcServer.handler=6,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:51:13,997 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134259856 > lower bound=16777216 2015-10-16 12:51:13,997 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:51:14,041 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630027_1099518554657] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 11214 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 765483, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:14,052 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 30939 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 544311, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:14,493 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630027_1099518554657] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 11787 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813618, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:14,493 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630028_1099518554658] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 588 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 961037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:14,493 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 32960 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 662785, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:14,494 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:51:14,494 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:51:14,494 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:51:14,494 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:51:14,494 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:51:14,885 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=5452659, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/0c24258685954b588d3113a9fa6e0fb1 2015-10-16 12:51:14,890 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/0c24258685954b588d3113a9fa6e0fb1 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0c24258685954b588d3113a9fa6e0fb1 2015-10-16 12:51:14,897 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0c24258685954b588d3113a9fa6e0fb1, entries=466290, sequenceid=5452659, filesize=70.8 M 2015-10-16 12:51:14,898 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134288240, currentsize=21.76 MB/22817440 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 901ms, sequenceid=5452659, compaction requested=true 2015-10-16 12:51:14,898 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:51:14,898 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 7 compacting, 2 eligible, 10 blocking 2015-10-16 12:51:14,899 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:51:14,899 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:51:14,899 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:51:14,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 35025 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 689683, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:14,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630027_1099518554657] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 12371 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1811252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:14,942 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:51:14,942 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:51:14,942 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:51:14,942 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:51:14,942 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:51:15,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630029_1099518554659] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 13083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 828528, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:15,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 37667 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 604172, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:15,462 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:15,462 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:15,462 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:15,462 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:15,462 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:15,654 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:51:15,687 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630021_1099518554651] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 38748 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 620280, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:15,694 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025067221 with entries=77594, filesize=121.63 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025075655 2015-10-16 12:51:15,695 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025059711 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025059711 2015-10-16 12:51:15,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 815868, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:16,021 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 1312 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 473131, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:16,021 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630029_1099518554659] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 13843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 737087, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:16,021 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:16,021 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:16,021 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:16,021 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:16,021 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:16,535 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:16,535 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:16,535 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630029_1099518554659] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 14476 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 642857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:16,535 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:16,536 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 4291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 456052, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:16,968 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 6436 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 61647508, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:16,968 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 12:51:16,968 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 62ms (threshold=20ms) 2015-10-16 12:51:16,968 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 62ms (threshold=20ms) 2015-10-16 12:51:16,968 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 62ms (threshold=20ms) 2015-10-16 12:51:16,968 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 62ms (threshold=20ms) 2015-10-16 12:51:17,087 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630031_1099518554661] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 15181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 777837, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:17,087 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 6922 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 443306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:17,087 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:17,087 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:17,087 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:17,087 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:17,599 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 9788 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 512417, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:17,599 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630031_1099518554661] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 15846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 700588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:17,599 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:17,599 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:17,599 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:17,599 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:17,600 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:18,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630031_1099518554661] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 16511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 725246, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:18,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 12632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 541013, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:18,129 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:18,129 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:18,129 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:18,129 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:18,129 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:18,476 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 14527 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 551548, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:18,662 DEBUG [PriorityRpcServer.handler=12,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:51:18,662 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134266544 > lower bound=16777216 2015-10-16 12:51:18,662 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:51:18,691 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 15707 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 480722, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:18,902 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 16871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 420767, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:18,902 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:18,902 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:18,902 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:18,902 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:18,902 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:19,528 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630032_1099518554662] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 17076 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839703, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:19,528 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 20484 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 511723, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:19,528 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630033_1099518554663] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 399 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 731522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:19,529 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:19,529 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:19,529 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:19,529 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:19,529 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:19,823 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 21361 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 677417, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:19,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630032_1099518554662] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 17724 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 796493, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:19,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630033_1099518554663] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 1048 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 796632, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:19,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 22096 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 773048, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:19,992 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:19,992 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:51:19,992 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:19,992 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:19,992 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:20,068 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=5499288, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/9a0d49c30d9644d397b6340e6adba373 2015-10-16 12:51:20,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/9a0d49c30d9644d397b6340e6adba373 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9a0d49c30d9644d397b6340e6adba373 2015-10-16 12:51:20,080 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9a0d49c30d9644d397b6340e6adba373, entries=466250, sequenceid=5499288, filesize=70.8 M 2015-10-16 12:51:20,081 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134275920, currentsize=37.56 MB/39382080 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1419ms, sequenceid=5499288, compaction requested=true 2015-10-16 12:51:20,081 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:51:20,081 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 7 compacting, 3 eligible, 10 blocking 2015-10-16 12:51:20,083 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222830447 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 12:51:20,083 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:51:20,083 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:51:20,083 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 12:51:20,083 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5fb2092b29984971b29aa5bceef5a2e9, keycount=466290, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=5406026 2015-10-16 12:51:20,084 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0c24258685954b588d3113a9fa6e0fb1, keycount=466290, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=5452659 2015-10-16 12:51:20,084 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9a0d49c30d9644d397b6340e6adba373, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=5499288 2015-10-16 12:51:20,100 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:51:20,392 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630034_1099518554664] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 336 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 694241, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:51:20,393 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630032_1099518554662] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 18198 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 737726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:20,393 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 23987 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 681324, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:20,393 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:20,782 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630032_1099518554662] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 18674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912257, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:20,782 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630034_1099518554664] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 883097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:51:21,543 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630034_1099518554664] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 2042 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 686170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:51:21,543 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 863ms (threshold=20ms); ack: seqno: 25537 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862504711, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:21,543 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 863ms (threshold=20ms) 2015-10-16 12:51:21,543 INFO [sync.0] wal.FSHLog: Slow sync cost: 863 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:21,543 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 863ms (threshold=20ms) 2015-10-16 12:51:21,543 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 863ms (threshold=20ms) 2015-10-16 12:51:21,543 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 862ms (threshold=20ms) 2015-10-16 12:51:21,543 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 863ms (threshold=20ms) 2015-10-16 12:51:21,543 INFO [sync.3] wal.FSHLog: Slow sync cost: 862 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:21,543 INFO [sync.1] wal.FSHLog: Slow sync cost: 862 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:21,543 INFO [sync.2] wal.FSHLog: Slow sync cost: 863 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:21,543 INFO [sync.4] wal.FSHLog: Slow sync cost: 862 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:21,801 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630036_1099518554666] hdfs.DFSClient: Slow ReadProcessor read fields took 201ms (threshold=20ms); ack: seqno: 2113 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 202657874, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:21,959 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630035_1099518554665] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 19358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819530, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:21,959 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 27582 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 431447, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:21,959 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630036_1099518554666] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 2372 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779877, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:21,959 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:21,959 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:21,959 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:21,959 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:21,959 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:51:22,329 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:51:22,329 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:51:22,330 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:51:22,329 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:51:22,329 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:51:22,329 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630036_1099518554666] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 2837 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794153, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:22,330 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630035_1099518554665] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 19812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017591, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:22,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630035_1099518554665] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 20268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 800115, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:22,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630036_1099518554666] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 3297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 772170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:22,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 31122 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 542228, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:22,699 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:22,699 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:22,699 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:22,699 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:22,699 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:22,813 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630036_1099518554666] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 3406 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868235, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:22,814 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#27 average throughput is 76.41 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 12:51:22,840 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/fafe4baf306a4939a566fe0606570530 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fafe4baf306a4939a566fe0606570530 2015-10-16 12:51:22,851 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:51:23,007 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 155ms (threshold=20ms) 2015-10-16 12:51:23,007 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 156ms (threshold=20ms) 2015-10-16 12:51:23,007 INFO [sync.1] wal.FSHLog: Slow sync cost: 155 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:23,007 INFO [sync.4] wal.FSHLog: Slow sync cost: 156 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:23,007 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 157ms (threshold=20ms) 2015-10-16 12:51:23,007 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 157ms (threshold=20ms) 2015-10-16 12:51:23,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630035_1099518554665] hdfs.DFSClient: Slow ReadProcessor read fields took 153ms (threshold=20ms); ack: seqno: 20451 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847250, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:23,007 INFO [sync.3] wal.FSHLog: Slow sync cost: 156 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:23,007 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 156ms (threshold=20ms) 2015-10-16 12:51:23,007 INFO [sync.2] wal.FSHLog: Slow sync cost: 156 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:23,007 INFO [sync.0] wal.FSHLog: Slow sync cost: 156 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:23,008 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:51:23,018 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5fb2092b29984971b29aa5bceef5a2e9, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5fb2092b29984971b29aa5bceef5a2e9 2015-10-16 12:51:23,026 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0c24258685954b588d3113a9fa6e0fb1, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0c24258685954b588d3113a9fa6e0fb1 2015-10-16 12:51:23,034 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9a0d49c30d9644d397b6340e6adba373, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9a0d49c30d9644d397b6340e6adba373 2015-10-16 12:51:23,035 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into fafe4baf306a4939a566fe0606570530(size=212.5 M), total size for store is 8.2 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-10-16 12:51:23,035 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=-1, time=36902988937112234; duration=2sec 2015-10-16 12:51:23,035 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:51:23,335 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 33481 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 504015, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:23,336 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:23,336 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:23,336 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:23,336 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:23,910 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630037_1099518554667] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 21427 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751909, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:23,910 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 36814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 495712, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:23,910 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:23,910 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:23,910 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:23,910 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:23,910 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:24,251 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:51:24,267 DEBUG [PriorityRpcServer.handler=16,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:51:24,267 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134281088 > lower bound=16777216 2015-10-16 12:51:24,267 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.03 MB 2015-10-16 12:51:24,283 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:51:24,295 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630030_1099518554660] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 39057 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 621419, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:24,302 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025075655 with entries=77760, filesize=121.89 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025084251 2015-10-16 12:51:24,304 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025067221 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025067221 2015-10-16 12:51:24,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1158115, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:24,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630039_1099518554669] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 822811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:51:24,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630037_1099518554667] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 21979 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 673243, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:24,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 681 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 535384, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:24,457 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:24,457 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:24,457 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:24,457 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:24,457 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:24,940 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:24,957 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:51:24,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630037_1099518554667] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 22447 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 821305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:24,957 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:51:24,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 2989 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 581829, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:24,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630039_1099518554669] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 828 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1018203, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:51:25,193 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=5545926, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b5ea6dcde76c40d59c0e884645a259a8 2015-10-16 12:51:25,205 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b5ea6dcde76c40d59c0e884645a259a8 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b5ea6dcde76c40d59c0e884645a259a8 2015-10-16 12:51:25,212 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b5ea6dcde76c40d59c0e884645a259a8, entries=466330, sequenceid=5545926, filesize=70.8 M 2015-10-16 12:51:25,214 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134298240, currentsize=24.35 MB/25536400 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 947ms, sequenceid=5545926, compaction requested=true 2015-10-16 12:51:25,214 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:51:25,214 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 7 compacting, 2 eligible, 10 blocking 2015-10-16 12:51:25,215 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:51:25,215 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:51:25,215 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:51:25,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 5596 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 641852, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:25,447 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:51:25,447 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:51:25,447 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:51:25,447 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:51:25,447 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:51:25,743 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 7063 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 488279, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:25,743 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 65ms (threshold=20ms) 2015-10-16 12:51:25,743 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 65ms (threshold=20ms) 2015-10-16 12:51:25,743 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 66ms (threshold=20ms) 2015-10-16 12:51:25,743 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 66ms (threshold=20ms) 2015-10-16 12:51:25,743 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 66ms (threshold=20ms) 2015-10-16 12:51:25,805 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630037_1099518554667] hdfs.DFSClient: Slow ReadProcessor read fields took 515ms (threshold=20ms); ack: seqno: 22922 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 884646, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:25,968 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 7721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905041, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:26,210 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630040_1099518554670] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 23514 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 740830, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:26,210 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:26,210 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 9004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 444849, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:26,210 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:26,210 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:26,210 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:26,211 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:26,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630040_1099518554670] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 24155 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 731986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:26,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 11895 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 423203, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:26,738 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:26,738 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:26,738 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:26,738 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:26,738 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:27,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 14849 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 421579, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:27,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630040_1099518554670] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 24801 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 820933, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:27,258 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:27,258 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:27,258 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:27,258 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:27,258 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:27,821 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630040_1099518554670] hdfs.DFSClient: Slow ReadProcessor read fields took 404ms (threshold=20ms); ack: seqno: 25006 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 391368448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:27,957 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:27,957 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:27,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630041_1099518554671] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 25196 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 817948, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:27,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 18818 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 428599, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:27,957 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:27,957 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:27,958 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:28,029 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 18880 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 671143, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:28,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 21515 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 545570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:28,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630041_1099518554671] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 25885 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833383, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:28,500 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:28,500 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:28,500 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:28,839 DEBUG [PriorityRpcServer.handler=13,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:51:28,839 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134267824 > lower bound=16777216 2015-10-16 12:51:28,840 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:51:28,866 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 23421 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 408550, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:29,012 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 24039 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 464950, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:29,012 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630042_1099518554672] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 151 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 892824, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:51:29,012 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:29,012 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:29,012 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630041_1099518554671] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 26541 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 881723, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:29,012 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:29,012 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:29,508 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630042_1099518554672] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 734456, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:51:29,509 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630043_1099518554673] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 27228 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898695, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:29,693 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=5592555, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b393d73d6f5443148f2587609a8e38f9 2015-10-16 12:51:29,699 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b393d73d6f5443148f2587609a8e38f9 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b393d73d6f5443148f2587609a8e38f9 2015-10-16 12:51:29,713 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b393d73d6f5443148f2587609a8e38f9, entries=466250, sequenceid=5592555, filesize=70.8 M 2015-10-16 12:51:29,754 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 472ms (threshold=20ms); ack: seqno: 25443 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 472092694, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:29,754 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 473ms (threshold=20ms) 2015-10-16 12:51:29,754 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 473ms (threshold=20ms) 2015-10-16 12:51:29,754 INFO [sync.2] wal.FSHLog: Slow sync cost: 472 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:29,755 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 472ms (threshold=20ms) 2015-10-16 12:51:29,754 INFO [sync.4] wal.FSHLog: Slow sync cost: 472 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:29,755 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 472ms (threshold=20ms) 2015-10-16 12:51:29,755 INFO [sync.3] wal.FSHLog: Slow sync cost: 472 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:29,755 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 473ms (threshold=20ms) 2015-10-16 12:51:29,755 INFO [sync.0] wal.FSHLog: Slow sync cost: 472 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:29,755 INFO [sync.1] wal.FSHLog: Slow sync cost: 473 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:29,756 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134275760, currentsize=11.12 MB/11654960 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 916ms, sequenceid=5592555, compaction requested=true 2015-10-16 12:51:29,756 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:51:29,756 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 7 compacting, 3 eligible, 10 blocking 2015-10-16 12:51:29,757 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:51:29,758 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:51:29,758 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:51:30,073 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 27160 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 526054, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:30,073 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630043_1099518554673] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 28004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 744234, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:30,073 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:51:30,073 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:51:30,073 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:51:30,073 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:51:30,073 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:51:30,589 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630043_1099518554673] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 28677 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 20201173, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:51:30,589 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 29968 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5714852, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:30,589 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:30,589 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:30,589 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:30,589 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:30,589 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:31,014 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630044_1099518554674] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 29175 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 67260156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:31,121 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 32357 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 875657, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:31,154 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 32390 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 480410, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:31,154 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630044_1099518554674] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 29434 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794379, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:31,154 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:31,154 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:31,154 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:31,154 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:31,154 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:31,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630044_1099518554674] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 30090 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813735, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:31,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 35310 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 519591, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:31,692 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:31,692 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:31,692 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:31,692 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:31,692 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:32,201 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630038_1099518554668] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 38075 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 523440, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:51:32,201 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630044_1099518554674] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 30770 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 746501, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:32,202 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:32,202 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:32,202 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:32,202 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:32,372 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:51:32,412 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:32,426 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025084251 with entries=78079, filesize=122.40 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025092373 2015-10-16 12:51:32,428 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025075655 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025075655 2015-10-16 12:51:32,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630046_1099518554676] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 31420 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 671594, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:32,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 1656 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 502225, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:32,728 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:32,728 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:32,728 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:32,728 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:32,728 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:33,247 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 4761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 918875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:33,247 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630046_1099518554676] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 32018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 686255, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:33,247 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:33,247 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:33,247 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:33,247 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:33,247 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:33,747 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630046_1099518554676] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 32680 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762253, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:33,747 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 7633 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 426175, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:33,747 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:33,747 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:33,747 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:33,747 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:33,748 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:33,759 DEBUG [PriorityRpcServer.handler=3,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:51:33,759 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:51:33,759 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:51:33,759 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 7 compacting, 3 eligible, 10 blocking 2015-10-16 12:51:33,761 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:51:33,761 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:51:33,761 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:51:34,293 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630046_1099518554676] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 33296 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 710282, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:34,293 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 10628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 398215, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:34,293 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:51:34,293 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:51:34,346 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630046_1099518554676] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 33297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 51249292, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:34,829 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630047_1099518554677] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 33920 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 765769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:34,829 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 13594 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 401607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:34,829 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:34,830 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:34,830 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:34,830 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:34,830 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:35,343 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 16458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 414291, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:35,343 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630047_1099518554677] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 34580 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 755857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:35,343 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:35,343 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:35,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 16656 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 795316, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:35,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 18793 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 595499, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:35,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630047_1099518554677] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 35368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 768800, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:35,913 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:35,913 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:35,913 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:35,913 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:35,913 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:36,427 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:51:36,427 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630048_1099518554678] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 36054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 805443, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:51:36,427 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:51:36,427 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 21554 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 473072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:36,427 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:51:36,427 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:51:36,427 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:51:36,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 24393 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 552271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:36,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630048_1099518554678] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 36721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767732, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:51:36,963 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:51:36,963 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:51:36,963 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:51:36,963 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:51:36,963 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:51:37,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630048_1099518554678] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 37466 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 774603, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:51:37,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 137ms (threshold=20ms); ack: seqno: 26893 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 107551150, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:37,533 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 137ms (threshold=20ms) 2015-10-16 12:51:37,533 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 138ms (threshold=20ms) 2015-10-16 12:51:37,533 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 137ms (threshold=20ms) 2015-10-16 12:51:37,533 INFO [sync.1] wal.FSHLog: Slow sync cost: 137 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:37,533 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 138ms (threshold=20ms) 2015-10-16 12:51:37,533 INFO [sync.4] wal.FSHLog: Slow sync cost: 137 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:37,533 INFO [sync.0] wal.FSHLog: Slow sync cost: 137 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:37,533 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 138ms (threshold=20ms) 2015-10-16 12:51:37,533 INFO [sync.3] wal.FSHLog: Slow sync cost: 137 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:37,533 INFO [sync.2] wal.FSHLog: Slow sync cost: 137 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:38,056 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 29758 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 597522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:38,056 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630049_1099518554679] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 38125 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729502, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:38,056 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:38,056 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:38,056 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:38,056 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:51:38,056 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:51:38,379 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 31155 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 504955, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:38,379 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630049_1099518554679] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 38483 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 752484, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:38,380 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 73ms (threshold=20ms) 2015-10-16 12:51:38,380 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 73ms (threshold=20ms) 2015-10-16 12:51:38,380 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 73ms (threshold=20ms) 2015-10-16 12:51:38,380 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 73ms (threshold=20ms) 2015-10-16 12:51:38,380 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 73ms (threshold=20ms) 2015-10-16 12:51:38,645 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630049_1099518554679] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 38815 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 849020, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:38,645 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 32434 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 810660, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:38,645 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:38,645 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:38,645 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:38,645 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:38,645 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 12:51:39,166 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:39,166 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 35253 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 569127, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:39,166 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630049_1099518554679] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 39494 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 747653, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:39,166 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:39,166 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:39,166 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:39,166 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:39,274 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630049_1099518554679] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 39594 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1120476, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:39,682 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630050_1099518554680] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 40159 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:39,682 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 38096 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 498965, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:39,682 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:39,682 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:39,682 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:39,682 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:39,682 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:40,004 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 39230 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 565535, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:40,005 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:51:40,270 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 40570 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 531496, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:40,270 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630050_1099518554680] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 40894 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839491, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:40,271 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:40,271 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:40,271 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:40,271 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:40,271 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:40,753 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630045_1099518554675] hdfs.DFSClient: Slow ReadProcessor read fields took 350ms (threshold=20ms); ack: seqno: 41302 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 347715919, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:40,756 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 739ms (threshold=20ms) 2015-10-16 12:51:40,767 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Slow sync cost: 749 ms, current pipeline: null 2015-10-16 12:51:40,948 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630050_1099518554680] hdfs.DFSClient: Slow ReadProcessor read fields took 169ms (threshold=20ms); ack: seqno: 41678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1000052, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:51:40,953 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 551ms (threshold=20ms) 2015-10-16 12:51:40,953 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 551ms (threshold=20ms) 2015-10-16 12:51:40,953 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 551ms (threshold=20ms) 2015-10-16 12:51:40,953 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 551ms (threshold=20ms) 2015-10-16 12:51:40,971 INFO [sync.4] wal.FSHLog: Slow sync cost: 568 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:40,953 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 551ms (threshold=20ms) 2015-10-16 12:51:40,971 INFO [sync.1] wal.FSHLog: Slow sync cost: 568 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:40,971 INFO [sync.2] wal.FSHLog: Slow sync cost: 568 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:40,971 INFO [sync.0] wal.FSHLog: Slow sync cost: 568 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:40,975 INFO [sync.3] wal.FSHLog: Slow sync cost: 573 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:40,993 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025092373 with entries=81665, filesize=128.02 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025100005 2015-10-16 12:51:40,997 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 240ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1112145, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:41,022 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630053_1099518554683] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 41680 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1477038, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:41,081 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 463063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:41,081 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630053_1099518554683] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 41800 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 725693, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:41,081 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:41,081 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:41,081 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:41,081 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:51:41,081 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:51:41,593 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:41,593 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:41,593 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 3176 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 420223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:41,593 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630053_1099518554683] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 42440 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 761571, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:41,593 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:51:41,593 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:42,104 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 6037 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 500088, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:42,104 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630053_1099518554683] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 43108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 679139, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:51:42,104 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:42,104 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:42,104 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:42,104 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:42,104 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:42,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 8832 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 435296, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:42,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630054_1099518554684] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 43776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2996107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:42,643 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 12:51:42,643 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 12:51:42,643 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 12:51:42,643 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 12:51:43,159 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 11618 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 552570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:43,159 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630054_1099518554684] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 44475 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 906135, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:43,160 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:51:43,160 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:51:43,160 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:51:43,160 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:51:43,160 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:51:43,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 14580 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 483134, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:43,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630054_1099518554684] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 45114 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 733956, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:43,671 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:43,672 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:43,672 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:44,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 17525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 511541, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:44,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630054_1099518554684] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 45725 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930206, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:51:44,174 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:44,174 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:44,174 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:44,174 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:44,174 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:44,665 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 20387 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 451001, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:44,665 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630055_1099518554685] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 46383 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 748627, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:51:44,665 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:44,665 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:44,665 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:44,665 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:44,665 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:51:45,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 23316 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 505041, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:45,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630055_1099518554685] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 47036 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 689110, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:51:45,164 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:45,164 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:45,164 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:45,164 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:45,164 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:45,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 25226 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 781688, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:45,717 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:45,717 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630055_1099518554685] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 47803 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 739362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:51:45,717 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 25738 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 466419, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:45,717 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:45,717 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:45,717 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:45,717 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:46,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 28605 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 445888, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:46,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630056_1099518554686] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 48454 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907219, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:51:46,235 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:46,235 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:46,235 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:46,235 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:46,235 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:51:46,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630056_1099518554686] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 49111 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 962329, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:51:46,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 31506 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 444896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:46,766 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:46,767 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:51:47,292 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 34478 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 502998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:47,292 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630056_1099518554686] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 49744 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 788447, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:51:47,293 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:51:48,087 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630057_1099518554687] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 50414 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 733996, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:48,947 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630057_1099518554687] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 51748 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 686906, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:49,815 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630058_1099518554688] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 53123 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 716909, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:51:50,656 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630059_1099518554689] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 54464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741629, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:51,518 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630059_1099518554689] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 55830 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 766605, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:51:52,346 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#26 average throughput is 72.84 MB/sec, slept 0 time(s) and total slept time is 0 ms. 0 active compactions remaining, total limit is unlimited 2015-10-16 12:51:52,375 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/81f31066d4f0409798a1d4a9948c5037 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/81f31066d4f0409798a1d4a9948c5037 2015-10-16 12:51:52,391 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 4636ms (threshold=20ms); ack: seqno: 37247 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917864, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:52,391 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:51:52,396 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:51:52,408 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eb253d4644d442638f3ed2235176bedd, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eb253d4644d442638f3ed2235176bedd 2015-10-16 12:51:52,416 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1b495f04e8814fdfb38eec4b2baacfec, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1b495f04e8814fdfb38eec4b2baacfec 2015-10-16 12:51:52,424 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3ec6ad24fd4d49e48a4cede70b168599, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3ec6ad24fd4d49e48a4cede70b168599 2015-10-16 12:51:52,433 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d0fe8512faa446e090b42ede97699a43, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d0fe8512faa446e090b42ede97699a43 2015-10-16 12:51:52,441 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c7e1590f6a494cf48a1a0b7ff48e8e97, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c7e1590f6a494cf48a1a0b7ff48e8e97 2015-10-16 12:51:52,449 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8990e31289164325a8923446ff5ab967, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8990e31289164325a8923446ff5ab967 2015-10-16 12:51:52,458 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fd6965ce2ece45309681b00bbe5bd53b, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fd6965ce2ece45309681b00bbe5bd53b 2015-10-16 12:51:52,458 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 7 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 81f31066d4f0409798a1d4a9948c5037(size=3.4 G), total size for store is 8.3 G. This selection was in queue for 0sec, and took 47sec to execute. 2015-10-16 12:51:52,458 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=7, fileSize=3.4 G, priority=2, time=36902974260805901; duration=47sec 2015-10-16 12:51:52,458 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:51:52,693 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536917456 > lower bound=16777216 2015-10-16 12:51:52,694 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=512.04 MB 2015-10-16 12:51:52,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 405ms (threshold=20ms); ack: seqno: 37248 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 802608, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:56,066 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=5778995, memsize=512.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/832e5c4ada5a487599575c458b72ce33 2015-10-16 12:51:56,073 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/832e5c4ada5a487599575c458b72ce33 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/832e5c4ada5a487599575c458b72ce33 2015-10-16 12:51:56,086 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/832e5c4ada5a487599575c458b72ce33, entries=1864350, sequenceid=5778995, filesize=283.2 M 2015-10-16 12:51:56,088 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 3292ms (threshold=20ms); ack: seqno: 37249 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 670054, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:51:56,088 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~512.04 MB/536917040, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 3395ms, sequenceid=5778995, compaction requested=true 2015-10-16 12:51:56,088 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:51:56,088 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 6 store files, 0 compacting, 6 eligible, 10 blocking 2015-10-16 12:51:56,091 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 6 files of size 9203341742 starting at candidate #0 after considering 10 permutations with 3 in ratio 2015-10-16 12:51:56,091 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction (all files) 2015-10-16 12:51:56,091 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 6 store files, 0 compacting, 6 eligible, 10 blocking 2015-10-16 12:51:56,094 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 6 files of size 9203341742 starting at candidate #0 after considering 10 permutations with 3 in ratio 2015-10-16 12:51:56,094 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction (all files) 2015-10-16 12:51:56,094 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:51:56,094 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 6 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=8.6 G 2015-10-16 12:51:56,095 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c01239c636ca4b4eae6973f4079040fd, keycount=30464060, bloomtype=ROW, size=4.5 G, encoding=NONE, seqNum=3046614, earliestPutTs=1445024396631 2015-10-16 12:51:56,095 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/81f31066d4f0409798a1d4a9948c5037, keycount=23126360, bloomtype=ROW, size=3.4 G, encoding=NONE, seqNum=5359393, earliestPutTs=1445024751055 2015-10-16 12:51:56,096 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fafe4baf306a4939a566fe0606570530, keycount=1398830, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=5499288, earliestPutTs=1445025064474 2015-10-16 12:51:56,096 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b5ea6dcde76c40d59c0e884645a259a8, keycount=466330, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=5545926, earliestPutTs=1445025078662 2015-10-16 12:51:56,097 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b393d73d6f5443148f2587609a8e38f9, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=5592555, earliestPutTs=1445025084267 2015-10-16 12:51:56,097 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/832e5c4ada5a487599575c458b72ce33, keycount=1864350, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=5778995, earliestPutTs=1445025088840 2015-10-16 12:51:56,145 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:51:56,515 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630064_1099518554694] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 526 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 791616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:51:57,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630064_1099518554694] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 1883 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:51:57,590 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630064_1099518554694] hdfs.DFSClient: Slow ReadProcessor read fields took 143ms (threshold=20ms); ack: seqno: 1891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937366, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:51:57,749 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630064_1099518554694] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 2048 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 867083, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:51:57,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630065_1099518554695] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 2214 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1140564, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:58,086 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630065_1099518554695] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 2371 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1136489, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:58,377 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630065_1099518554695] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 2693 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1363929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:58,556 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630065_1099518554695] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 2848 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012864, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:58,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630065_1099518554695] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 3004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1208466, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:58,920 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630065_1099518554695] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 3156 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1076328, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:59,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630065_1099518554695] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 3251 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914994, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:59,069 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630065_1099518554695] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 3257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026699, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:59,243 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630065_1099518554695] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 3414 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1140480, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:59,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630065_1099518554695] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 3576 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1030129, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:59,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630065_1099518554695] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 3728 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 999123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:51:59,758 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630065_1099518554695] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 3877 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1091478, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:52:00,421 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 163ms (threshold=20ms); ack: seqno: 4255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1036879, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:00,485 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 4263 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1419652, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:00,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 4426 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984091, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:00,817 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 4575 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1093668, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:00,877 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 4616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 760606, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:00,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 4622 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047219, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:01,121 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 4775 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1172029, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:01,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 4927 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1168736, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:01,482 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 5078 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1230238, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:01,668 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 5224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:01,869 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 5375 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1136624, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:02,047 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 5533 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1016481, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:02,160 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 5648 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 34104033, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:02,276 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 5683 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794933, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:02,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 5845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845647, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:02,548 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 5978 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 722167, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:02,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 5985 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909626, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:02,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630066_1099518554696] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 6137 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1083513, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:02,959 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 6292 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1315409, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:03,138 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 6448 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912370, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:03,333 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 6602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932862, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:03,516 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 6765 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 882166, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:03,695 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 6921 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1051803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:03,907 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 7081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887402, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:04,085 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 7238 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 981579, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:04,188 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 7353 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 663655, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:04,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 7363 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005086, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:04,538 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 7495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1052942, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:04,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 7634 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794987, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:04,859 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 7774 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2140477, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:05,231 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 294ms (threshold=20ms); ack: seqno: 7886 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1258458, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:05,285 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 7973 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 796519, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:05,456 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 8131 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038030, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:05,626 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630067_1099518554697] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 8266 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 957764, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:52:05,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 9844ms (threshold=20ms); ack: seqno: 37250 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866628, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:05,998 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 37256 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819476, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:06,122 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 37258 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 958698, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:06,312 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 188ms (threshold=20ms); ack: seqno: 37270 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844332, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:06,445 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 37273 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3480583, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:06,511 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 37275 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 629342, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:06,598 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630068_1099518554698] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 8661 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 788127, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:06,599 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 37290 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 673268, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:06,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 37291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 619558, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:06,812 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 137ms (threshold=20ms); ack: seqno: 37300 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 662720, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:06,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 37309 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769236, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:07,072 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 37319 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 738963, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:07,148 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 37329 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873260, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:07,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 37344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 688090, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:07,279 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 37359 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 460926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:07,493 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630068_1099518554698] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 10007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 742953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:07,494 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 210ms (threshold=20ms); ack: seqno: 37392 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1166917, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:07,841 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:52:08,122 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 40910 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 531792, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:08,122 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:52:08,122 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:52:08,122 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:52:08,122 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:52:08,122 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:52:09,829 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630051_1099518554681] hdfs.DFSClient: Slow ReadProcessor read fields took 1629ms (threshold=20ms); ack: seqno: 41368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2944675, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:09,835 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 1987ms (threshold=20ms) 2015-10-16 12:52:09,848 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Slow sync cost: 1999 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:09,867 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630071_1099518554701] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 41370 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 22705421, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:09,867 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 1668ms (threshold=20ms) 2015-10-16 12:52:09,867 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 1668ms (threshold=20ms) 2015-10-16 12:52:09,867 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 1668ms (threshold=20ms) 2015-10-16 12:52:09,867 INFO [sync.0] wal.FSHLog: Slow sync cost: 1668 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:09,867 INFO [sync.2] wal.FSHLog: Slow sync cost: 1668 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:09,867 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 1668ms (threshold=20ms) 2015-10-16 12:52:09,868 INFO [sync.4] wal.FSHLog: Slow sync cost: 1668 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:09,867 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 1668ms (threshold=20ms) 2015-10-16 12:52:09,868 INFO [sync.1] wal.FSHLog: Slow sync cost: 1668 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:09,883 INFO [sync.3] wal.FSHLog: Slow sync cost: 1682 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:09,883 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025100005 with entries=81664, filesize=128.02 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025127841 2015-10-16 12:52:09,885 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025084251 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025084251 2015-10-16 12:52:09,886 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630070_1099518554700] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 978761, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:09,890 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025092373 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025092373 2015-10-16 12:52:09,915 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 11597 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874824, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:09,976 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630070_1099518554700] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 338 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 467174, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:09,976 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:09,976 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:09,976 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:09,976 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:09,976 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:09,977 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 11632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 881863, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:10,130 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 146ms (threshold=20ms); ack: seqno: 11639 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1082642, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:10,288 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 11787 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 923366, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:10,538 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 203ms (threshold=20ms); ack: seqno: 11857 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1051767, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:10,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 11864 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1083798, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:10,721 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 11937 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812186, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:10,721 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630070_1099518554700] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 4605 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 550215, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:10,721 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:10,721 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:10,721 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:10,721 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:10,721 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:10,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 11944 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 831527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:11,012 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 12077 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1059430, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:11,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 286ms (threshold=20ms); ack: seqno: 12084 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1338400, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:11,432 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 12092 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1151569, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:11,566 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630070_1099518554700] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 9416 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 401972, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:11,566 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:11,566 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:11,566 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:11,566 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:11,566 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:11,620 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 182ms (threshold=20ms); ack: seqno: 12100 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 10336531, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:11,707 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630070_1099518554700] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 9556 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 601333, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:11,812 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 12255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013568, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:11,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 12263 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:12,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 279ms (threshold=20ms); ack: seqno: 12270 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1232842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:12,321 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 12277 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 977099, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:12,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630070_1099518554700] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 14202 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 560851, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:12,530 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:12,530 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:12,530 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:12,530 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:12,668 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 344ms (threshold=20ms); ack: seqno: 12284 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1282449, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:12,774 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 12292 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1264919, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:12,886 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 12299 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868490, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:13,186 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 295ms (threshold=20ms); ack: seqno: 12308 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1287747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:13,295 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 12315 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1302151, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:13,336 DEBUG [PriorityRpcServer.handler=3,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:52:13,337 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134272560 > lower bound=16777216 2015-10-16 12:52:13,337 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:52:13,366 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630070_1099518554700] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 19168 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 513577, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:13,448 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630072_1099518554702] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 64 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 776650, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:13,448 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630070_1099518554700] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 19444 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 543223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:13,448 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:13,448 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:13,448 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:13,448 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:13,448 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:13,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 302ms (threshold=20ms); ack: seqno: 12323 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1433855, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:13,709 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 12330 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1307901, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:13,985 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 273ms (threshold=20ms); ack: seqno: 12338 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1386738, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:14,116 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630070_1099518554700] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 23169 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 435804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:14,116 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630072_1099518554702] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 988 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2396789, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:14,116 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:14,116 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:14,116 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:14,116 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:14,116 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:14,118 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 12346 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1415986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:14,223 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=5825624, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/feec5ec949414e7d8bf0837af0cc695a 2015-10-16 12:52:14,229 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/feec5ec949414e7d8bf0837af0cc695a as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/feec5ec949414e7d8bf0837af0cc695a 2015-10-16 12:52:14,234 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/feec5ec949414e7d8bf0837af0cc695a, entries=466250, sequenceid=5825624, filesize=70.8 M 2015-10-16 12:52:14,236 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134275600, currentsize=25.75 MB/27002000 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 899ms, sequenceid=5825624, compaction requested=false 2015-10-16 12:52:14,390 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 268ms (threshold=20ms); ack: seqno: 12354 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1176830, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:14,510 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 12364 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1065269, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:14,806 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 293ms (threshold=20ms); ack: seqno: 12371 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1321032, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:14,920 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 12379 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1398724, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:15,009 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630070_1099518554700] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 28250 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 417097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:15,009 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:15,009 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:15,010 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:15,010 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:15,010 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:15,210 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 285ms (threshold=20ms); ack: seqno: 12386 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1248261, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:15,322 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 12394 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1181094, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:15,618 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 291ms (threshold=20ms); ack: seqno: 12401 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1216295, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:15,748 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 12408 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1243251, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:15,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630070_1099518554700] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 33719 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 464585, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:15,938 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:15,938 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:15,938 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:52:15,938 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:52:15,938 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:52:16,036 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 285ms (threshold=20ms); ack: seqno: 12415 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1251529, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:16,151 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630070_1099518554700] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 34210 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 649798, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:16,940 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630070_1099518554700] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 38631 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 414327, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:16,964 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 56ms (threshold=20ms) 2015-10-16 12:52:16,964 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 56ms (threshold=20ms) 2015-10-16 12:52:16,964 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 55ms (threshold=20ms) 2015-10-16 12:52:16,964 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630069_1099518554699] hdfs.DFSClient: Slow ReadProcessor read fields took 863ms (threshold=20ms); ack: seqno: 12502 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865412, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:16,964 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 55ms (threshold=20ms) 2015-10-16 12:52:16,964 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 56ms (threshold=20ms) 2015-10-16 12:52:17,000 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:52:17,242 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 12587 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841876, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:17,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 12596 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1015425, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:17,372 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 365ms (threshold=20ms) 2015-10-16 12:52:17,381 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Slow sync cost: 374 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:17,390 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025127841 with entries=81863, filesize=128.33 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025137000 2015-10-16 12:52:17,392 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025100005 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025100005 2015-10-16 12:52:17,393 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630074_1099518554704] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 840712, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:52:17,472 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 12604 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060395, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:17,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 12612 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 960816, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:17,642 DEBUG [PriorityRpcServer.handler=19,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:52:17,642 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134253056 > lower bound=16777216 2015-10-16 12:52:17,642 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:52:17,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630074_1099518554704] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 1528 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 505476, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:52:17,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 12620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842780, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:17,811 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 12628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985028, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:17,853 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630076_1099518554706] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839131, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:52:17,853 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630074_1099518554704] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 2458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 390083, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:52:17,853 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:17,853 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:17,853 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:17,853 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:17,853 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:17,929 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 12636 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927455, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:18,052 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 12643 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1016282, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:18,089 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 12651 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934583, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:18,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 12658 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:18,297 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 172ms (threshold=20ms); ack: seqno: 12666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1114524, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:18,473 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=5872251, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8368eb7df7ac48a8bf083135b783dc2c 2015-10-16 12:52:18,478 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8368eb7df7ac48a8bf083135b783dc2c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8368eb7df7ac48a8bf083135b783dc2c 2015-10-16 12:52:18,484 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8368eb7df7ac48a8bf083135b783dc2c, entries=466230, sequenceid=5872251, filesize=70.8 M 2015-10-16 12:52:18,513 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630074_1099518554704] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 6115 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 536796, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:52:18,513 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:18,513 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:18,513 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:18,514 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134269920, currentsize=25.30 MB/26526880 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 872ms, sequenceid=5872251, compaction requested=false 2015-10-16 12:52:18,550 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 249ms (threshold=20ms); ack: seqno: 12672 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1097872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:19,026 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 473ms (threshold=20ms); ack: seqno: 12680 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1636986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:19,179 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 148ms (threshold=20ms); ack: seqno: 12688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154987, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:19,333 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 12703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066341, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:19,359 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 12708 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986829, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:19,463 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630074_1099518554704] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 11467 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 400628, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:52:19,463 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:19,463 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:19,463 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:19,463 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:19,463 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:19,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 224ms (threshold=20ms); ack: seqno: 12716 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1325872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:19,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 12724 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1278926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:19,759 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 12733 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1159712, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:20,039 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 254ms (threshold=20ms); ack: seqno: 12750 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1203072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:20,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 12757 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111479, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:20,237 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 12773 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909113, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:20,264 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 12780 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859738, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:20,374 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 12787 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950224, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:20,420 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 12797 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895284, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:20,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 12807 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967525, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:20,466 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630074_1099518554704] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 16702 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 682626, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:52:20,490 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 12814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989832, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:20,710 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 12957 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1077879, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:20,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 12981 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926977, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:20,944 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 12990 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912030, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:21,056 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 12998 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:21,264 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630074_1099518554704] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 20830 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 508025, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:52:21,264 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:21,264 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:21,264 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:21,264 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:21,352 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 152ms (threshold=20ms); ack: seqno: 13158 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948914, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:21,539 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 157ms (threshold=20ms); ack: seqno: 13173 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 875708, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:21,574 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 13180 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:21,720 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 141ms (threshold=20ms); ack: seqno: 13189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026914, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:21,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 140ms (threshold=20ms); ack: seqno: 13196 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 955066, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:21,960 DEBUG [PriorityRpcServer.handler=12,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:52:21,961 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134277760 > lower bound=16777216 2015-10-16 12:52:21,971 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.36 MB 2015-10-16 12:52:22,000 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630074_1099518554704] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 25000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 471738, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:52:22,016 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 13211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062288, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:22,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630074_1099518554704] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 25764 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 545788, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:52:22,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630077_1099518554707] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 176 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 725822, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:22,171 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:22,171 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:22,171 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:22,171 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:22,171 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:22,379 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 355ms (threshold=20ms); ack: seqno: 13218 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060140, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:22,439 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 13226 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:22,767 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 322ms (threshold=20ms); ack: seqno: 13234 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1187073, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:22,806 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=5919010, memsize=128.4 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ab826cc41ae745dc969a2f7d8c543a6b 2015-10-16 12:52:22,811 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ab826cc41ae745dc969a2f7d8c543a6b as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ab826cc41ae745dc969a2f7d8c543a6b 2015-10-16 12:52:22,817 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ab826cc41ae745dc969a2f7d8c543a6b, entries=467550, sequenceid=5919010, filesize=71.0 M 2015-10-16 12:52:22,818 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.41 MB/134649440, currentsize=23.46 MB/24600000 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 847ms, sequenceid=5919010, compaction requested=true 2015-10-16 12:52:22,818 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:52:22,819 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 6 compacting, 3 eligible, 10 blocking 2015-10-16 12:52:22,820 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 223020215 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 12:52:22,820 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:52:22,820 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:52:22,820 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.7 M 2015-10-16 12:52:22,820 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/feec5ec949414e7d8bf0837af0cc695a, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=5825624 2015-10-16 12:52:22,820 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8368eb7df7ac48a8bf083135b783dc2c, keycount=466230, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=5872251 2015-10-16 12:52:22,820 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ab826cc41ae745dc969a2f7d8c543a6b, keycount=467550, bloomtype=ROW, size=71.0 M, encoding=NONE, seqNum=5919010 2015-10-16 12:52:22,833 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:52:22,868 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630074_1099518554704] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 29277 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 598638, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:52:22,868 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:22,868 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:22,868 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:22,868 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:22,868 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:22,935 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 165ms (threshold=20ms); ack: seqno: 13242 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1315594, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:22,980 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 13243 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 46303146, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:23,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 189ms (threshold=20ms); ack: seqno: 13251 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 894758, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:23,421 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630074_1099518554704] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 32209 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 601645, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:52:23,421 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630078_1099518554708] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 702008, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:23,421 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:23,422 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:23,422 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:23,421 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:23,421 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:23,579 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 303ms (threshold=20ms); ack: seqno: 13385 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1318707, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:23,698 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 13393 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023166, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:23,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630078_1099518554708] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1216 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 778813, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:23,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630074_1099518554704] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 34747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 609243, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:52:23,941 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:23,941 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:23,941 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:23,941 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:23,941 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:24,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 300ms (threshold=20ms); ack: seqno: 13400 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1136813, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:24,108 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 13408 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1136172, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:24,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 286ms (threshold=20ms); ack: seqno: 13416 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127179, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:24,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630074_1099518554704] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 37281 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 609274, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:52:24,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630078_1099518554708] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 1908 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:24,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 13441 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 840646, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:24,443 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:24,443 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:24,443 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:24,443 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:24,443 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:24,539 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 13446 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:24,725 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:52:24,756 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025137000 with entries=77795, filesize=121.95 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025144726 2015-10-16 12:52:24,758 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025127841 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025127841 2015-10-16 12:52:24,848 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 302ms (threshold=20ms); ack: seqno: 13454 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1212886, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:24,945 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630080_1099518554710] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 828469, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:24,972 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 13461 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1129582, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:25,000 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630080_1099518554710] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 442 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 502747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:25,000 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 13463 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1035160, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:25,000 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630079_1099518554709] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 2730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 749540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:52:25,000 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:25,000 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:25,000 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:25,000 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:25,000 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:25,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 275ms (threshold=20ms); ack: seqno: 13468 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1449470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:25,472 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630079_1099518554709] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 3389 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701703, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:52:25,472 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630080_1099518554710] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 2749 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 462793, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:25,472 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:25,472 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:25,472 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:25,472 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:25,518 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#29 average throughput is 77.19 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 12:52:25,535 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/e011799159764a9683daebab3bc6f669 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e011799159764a9683daebab3bc6f669 2015-10-16 12:52:25,543 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:52:25,546 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:52:25,555 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/feec5ec949414e7d8bf0837af0cc695a, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/feec5ec949414e7d8bf0837af0cc695a 2015-10-16 12:52:25,564 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8368eb7df7ac48a8bf083135b783dc2c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8368eb7df7ac48a8bf083135b783dc2c 2015-10-16 12:52:25,572 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ab826cc41ae745dc969a2f7d8c543a6b, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ab826cc41ae745dc969a2f7d8c543a6b 2015-10-16 12:52:25,572 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into e011799159764a9683daebab3bc6f669(size=212.7 M), total size for store is 8.8 G. This selection was in queue for 0sec, and took 2sec to execute. 2015-10-16 12:52:25,572 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.7 M, priority=2, time=36903051674024106; duration=2sec 2015-10-16 12:52:25,573 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:52:25,724 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 342ms (threshold=20ms); ack: seqno: 13598 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1204576, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:25,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 13606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1105553, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:26,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 266ms (threshold=20ms); ack: seqno: 13613 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1107324, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:26,229 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 13621 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1104545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:26,329 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630080_1099518554710] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 7479 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 535961, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:26,329 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 13724 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 781848, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:26,330 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:26,330 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:26,330 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:26,330 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:26,330 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:26,383 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 13731 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 886419, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:26,602 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 13878 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 860241, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:26,649 DEBUG [PriorityRpcServer.handler=0,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:52:26,649 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134253584 > lower bound=16777216 2015-10-16 12:52:26,649 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:52:26,676 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630080_1099518554710] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 9286 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 660293, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:26,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 292ms (threshold=20ms); ack: seqno: 13886 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1155537, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:27,008 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 13894 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:27,065 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:27,065 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630080_1099518554710] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 11232 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 463489, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:27,065 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630081_1099518554711] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 533 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 810197, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:52:27,065 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:27,065 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:27,065 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:27,066 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:27,290 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 277ms (threshold=20ms); ack: seqno: 13902 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1267925, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:27,397 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 13909 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1123186, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:27,464 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=5965640, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/2529360c9663442e921e83a9c263954e 2015-10-16 12:52:27,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/2529360c9663442e921e83a9c263954e as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2529360c9663442e921e83a9c263954e 2015-10-16 12:52:27,475 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2529360c9663442e921e83a9c263954e, entries=466250, sequenceid=5965640, filesize=70.8 M 2015-10-16 12:52:27,478 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134275920, currentsize=22.24 MB/23325040 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 829ms, sequenceid=5965640, compaction requested=false 2015-10-16 12:52:27,703 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 300ms (threshold=20ms); ack: seqno: 13917 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1374035, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:27,814 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 13925 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1221109, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:27,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630080_1099518554710] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 15474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 591805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:27,879 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:27,879 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:27,879 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:27,879 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:27,879 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:27,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 13933 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1251951, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:28,107 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 14082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:28,320 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 14219 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:28,618 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 293ms (threshold=20ms); ack: seqno: 14226 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1195858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:28,711 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630080_1099518554710] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 19873 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 471991, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:28,711 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:28,711 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:28,711 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:28,711 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:28,712 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:28,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 14234 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993200, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:29,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 284ms (threshold=20ms); ack: seqno: 14241 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069480, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:29,151 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 14249 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1175300, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:29,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 288ms (threshold=20ms); ack: seqno: 14256 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043175, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:29,560 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 14265 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1246766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:29,652 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630080_1099518554710] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 25296 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 555426, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:29,652 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:29,652 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:29,652 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:29,652 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:29,653 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:29,854 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 290ms (threshold=20ms); ack: seqno: 14273 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154120, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:29,971 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 14281 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5610790, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:30,134 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630080_1099518554710] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 27464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 618759, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:30,273 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 301ms (threshold=20ms); ack: seqno: 14290 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1313001, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:30,425 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 147ms (threshold=20ms); ack: seqno: 14297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1189096, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:30,703 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630080_1099518554710] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 30785 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 408005, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:30,704 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:52:30,704 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:30,704 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:30,705 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 276ms (threshold=20ms); ack: seqno: 14305 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1059176, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:30,821 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 14313 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1209271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:31,001 DEBUG [PriorityRpcServer.handler=16,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:52:31,002 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134281936 > lower bound=16777216 2015-10-16 12:52:31,002 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:52:31,099 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 274ms (threshold=20ms); ack: seqno: 14320 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1360311, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:31,217 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 14328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1165753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:31,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630080_1099518554710] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 34957 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 487323, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:31,447 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:52:31,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630082_1099518554712] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 627 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 713364, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:31,447 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:52:31,447 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:52:31,447 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:52:31,447 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:52:31,553 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 331ms (threshold=20ms); ack: seqno: 14336 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1334953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:31,716 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 14457 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1140666, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:31,772 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=6012271, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d4ecafb5ab5547b5b1f99629cfb4b736 2015-10-16 12:52:31,777 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d4ecafb5ab5547b5b1f99629cfb4b736 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d4ecafb5ab5547b5b1f99629cfb4b736 2015-10-16 12:52:31,783 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d4ecafb5ab5547b5b1f99629cfb4b736, entries=466270, sequenceid=6012271, filesize=70.8 M 2015-10-16 12:52:31,785 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134281520, currentsize=21.75 MB/22808800 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 783ms, sequenceid=6012271, compaction requested=true 2015-10-16 12:52:31,785 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:52:31,785 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 6 compacting, 3 eligible, 10 blocking 2015-10-16 12:52:31,786 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:52:31,786 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:52:31,786 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:52:32,062 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630080_1099518554710] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 38291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 515764, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:32,071 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:52:32,071 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:52:32,071 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:52:32,071 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:52:32,071 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:52:32,169 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:52:32,988 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630073_1099518554703] hdfs.DFSClient: Slow ReadProcessor read fields took 1179ms (threshold=20ms); ack: seqno: 14586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1058183, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:32,988 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630080_1099518554710] hdfs.DFSClient: Slow ReadProcessor read fields took 478ms (threshold=20ms); ack: seqno: 40943 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2247208, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:32,993 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 813ms (threshold=20ms) 2015-10-16 12:52:33,022 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Slow sync cost: 841 ms, current pipeline: null 2015-10-16 12:52:33,404 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 895ms (threshold=20ms) 2015-10-16 12:52:33,404 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 894ms (threshold=20ms) 2015-10-16 12:52:33,404 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 894ms (threshold=20ms) 2015-10-16 12:52:33,404 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 894ms (threshold=20ms) 2015-10-16 12:52:33,404 INFO [sync.1] wal.FSHLog: Slow sync cost: 893 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:33,404 INFO [sync.4] wal.FSHLog: Slow sync cost: 893 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:33,404 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 895ms (threshold=20ms) 2015-10-16 12:52:33,404 INFO [sync.2] wal.FSHLog: Slow sync cost: 893 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:33,404 INFO [sync.3] wal.FSHLog: Slow sync cost: 894 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:33,409 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 165ms (threshold=20ms); ack: seqno: 14957 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:33,443 INFO [sync.0] wal.FSHLog: Slow sync cost: 932 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:33,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 14965 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827459, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:33,444 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025144726 with entries=81670, filesize=128.02 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025152169 2015-10-16 12:52:33,446 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025137000 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025137000 2015-10-16 12:52:33,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630083_1099518554713] hdfs.DFSClient: Slow ReadProcessor read fields took 454ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1218901, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:33,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 14972 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928377, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:33,632 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 14980 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974126, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:33,814 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 177ms (threshold=20ms); ack: seqno: 14987 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062899, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:34,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 15122 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922637, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:34,292 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630083_1099518554713] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 4891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489991, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:34,292 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:34,292 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:34,292 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:34,292 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:34,292 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:34,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 278ms (threshold=20ms); ack: seqno: 15129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1292558, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:34,404 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 15137 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1136575, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:34,687 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 279ms (threshold=20ms); ack: seqno: 15144 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1465577, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:34,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 15152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020455, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:35,107 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 302ms (threshold=20ms); ack: seqno: 15160 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1240877, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:35,256 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630083_1099518554713] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 10319 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 481260, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:35,256 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 15168 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1119546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:35,256 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:35,256 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:35,256 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:35,256 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:35,256 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:35,554 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 292ms (threshold=20ms); ack: seqno: 15176 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1286803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:35,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 15184 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 998043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:35,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 247ms (threshold=20ms); ack: seqno: 15192 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1316970, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:35,960 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 15200 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1187374, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:36,058 DEBUG [PriorityRpcServer.handler=19,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:52:36,058 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134268240 > lower bound=16777216 2015-10-16 12:52:36,058 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:52:36,073 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 15209 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1100267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:36,088 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630083_1099518554713] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 15244 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 518160, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:36,150 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 15254 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 718564, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:36,150 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630083_1099518554713] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 15402 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:36,151 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:36,151 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:36,151 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:36,151 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:36,151 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:36,206 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 15260 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985485, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:36,382 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 15388 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895446, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:36,559 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 15511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 900789, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:36,710 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630086_1099518554716] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 629 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 705420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:52:36,710 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630083_1099518554713] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 18240 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 548531, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:36,710 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:36,710 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:36,710 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:36,710 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:36,761 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 15640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865568, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:36,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 15769 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 890398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:37,016 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630083_1099518554713] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 19230 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767493, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:37,047 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=6058900, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/175cce2eb47d419f925f8f6d13169ccd 2015-10-16 12:52:37,056 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/175cce2eb47d419f925f8f6d13169ccd as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/175cce2eb47d419f925f8f6d13169ccd 2015-10-16 12:52:37,067 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/175cce2eb47d419f925f8f6d13169ccd, entries=466250, sequenceid=6058900, filesize=70.8 M 2015-10-16 12:52:37,068 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134275600, currentsize=23.55 MB/24697920 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1010ms, sequenceid=6058900, compaction requested=true 2015-10-16 12:52:37,093 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:52:37,093 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 6 compacting, 4 eligible, 10 blocking 2015-10-16 12:52:37,095 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 445840791 starting at candidate #0 after considering 3 permutations with 2 in ratio 2015-10-16 12:52:37,095 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:52:37,096 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:52:37,096 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=425.2 M 2015-10-16 12:52:37,096 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e011799159764a9683daebab3bc6f669, keycount=1400030, bloomtype=ROW, size=212.7 M, encoding=NONE, seqNum=5919010 2015-10-16 12:52:37,096 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2529360c9663442e921e83a9c263954e, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=5965640 2015-10-16 12:52:37,096 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d4ecafb5ab5547b5b1f99629cfb4b736, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=6012271 2015-10-16 12:52:37,096 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/175cce2eb47d419f925f8f6d13169ccd, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=6058900 2015-10-16 12:52:37,117 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:52:37,254 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630087_1099518554717] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 111 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 697574, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:37,254 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 16130 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 765628, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:37,254 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630083_1099518554713] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 20245 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 544443, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:37,254 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:37,254 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:37,254 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:37,254 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:37,254 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:37,649 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630087_1099518554717] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 526 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 743026, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:37,649 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630084_1099518554714] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 16608 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 843477, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:37,649 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:37,649 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:37,649 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:37,649 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:37,649 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:38,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630087_1099518554717] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 895 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022491, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:38,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630083_1099518554713] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 23776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 719839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:38,066 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 78ms (threshold=20ms) 2015-10-16 12:52:38,066 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 78ms (threshold=20ms) 2015-10-16 12:52:38,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630088_1099518554718] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 17127 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 931567, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:38,069 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 80ms (threshold=20ms) 2015-10-16 12:52:38,069 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 81ms (threshold=20ms) 2015-10-16 12:52:38,069 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 80ms (threshold=20ms) 2015-10-16 12:52:38,474 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:52:38,474 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:52:38,474 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630087_1099518554717] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 1381 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 877239, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:38,474 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:52:38,474 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:52:38,474 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630088_1099518554718] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 17622 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736973, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:38,474 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:52:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.0, tune compaction throughput to 20.00 MB/sec 2015-10-16 12:52:38,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630083_1099518554713] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 27357 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 603335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:38,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630087_1099518554717] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 1812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986008, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:38,875 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:52:38,876 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:52:38,876 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:52:38,875 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:52:38,875 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:52:38,877 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630088_1099518554718] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 18104 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 933944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:39,740 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630083_1099518554713] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 32259 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 469224, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:39,740 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:39,740 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:39,740 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:39,740 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:39,740 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:39,743 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630087_1099518554717] hdfs.DFSClient: Slow ReadProcessor read fields took 849ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1641742, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:39,830 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630088_1099518554718] hdfs.DFSClient: Slow ReadProcessor read fields took 845ms (threshold=20ms); ack: seqno: 18243 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1179717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:40,518 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630083_1099518554713] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 36479 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 384837, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:40,518 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:40,518 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:40,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630087_1099518554717] hdfs.DFSClient: Slow ReadProcessor read fields took 865ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1375645, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:52:40,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630088_1099518554718] hdfs.DFSClient: Slow ReadProcessor read fields took 877ms (threshold=20ms); ack: seqno: 18410 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1097057, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:40,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630088_1099518554718] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 18416 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1098022, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:40,891 DEBUG [PriorityRpcServer.handler=8,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:52:40,891 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134276176 > lower bound=16777216 2015-10-16 12:52:40,891 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:52:40,920 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630083_1099518554713] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 38567 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 785936, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:40,988 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:52:41,059 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630088_1099518554718] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 18570 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888547, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:41,292 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630083_1099518554713] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 40484 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 606957, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:41,292 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:52:41,292 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:52:41,292 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:52:41,292 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:52:41,292 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:52:41,332 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 336ms (threshold=20ms) 2015-10-16 12:52:41,338 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Slow sync cost: 341 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:41,355 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025152169 with entries=81159, filesize=127.22 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025160988 2015-10-16 12:52:41,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630091_1099518554721] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888481, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:52:41,359 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025144726 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025144726 2015-10-16 12:52:41,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 857ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1491392, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:41,830 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630088_1099518554718] hdfs.DFSClient: Slow ReadProcessor read fields took 767ms (threshold=20ms); ack: seqno: 18577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1355163, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:41,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630088_1099518554718] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 18623 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 749955, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:41,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630091_1099518554721] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 2977 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 577354, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:52:41,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 2295 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1383111, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:41,899 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:41,899 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:41,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630090_1099518554720] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 897 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 781702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:52:41,900 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:41,900 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:41,899 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:42,130 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=6105534, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/bb87dca51ce74d4e8934761f39c8bcec 2015-10-16 12:52:42,142 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/bb87dca51ce74d4e8934761f39c8bcec as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bb87dca51ce74d4e8934761f39c8bcec 2015-10-16 12:52:42,150 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bb87dca51ce74d4e8934761f39c8bcec, entries=466300, sequenceid=6105534, filesize=70.8 M 2015-10-16 12:52:42,151 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134290160, currentsize=35.89 MB/37634640 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1260ms, sequenceid=6105534, compaction requested=false 2015-10-16 12:52:42,677 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630091_1099518554721] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 7373 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 425771, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:52:42,678 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:42,678 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:42,678 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:52:42,678 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:42,678 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:52:42,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 785ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1376534, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:42,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630088_1099518554718] hdfs.DFSClient: Slow ReadProcessor read fields took 841ms (threshold=20ms); ack: seqno: 18744 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1367004, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:43,224 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630091_1099518554721] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 9756 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 798456, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:52:43,572 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630091_1099518554721] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 11464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 561620, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:52:43,572 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:43,572 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:43,572 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:43,572 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:52:43,572 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:43,743 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 829ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2684071, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:43,829 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630092_1099518554722] hdfs.DFSClient: Slow ReadProcessor read fields took 869ms (threshold=20ms); ack: seqno: 18914 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1125544, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:44,322 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630091_1099518554721] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 15722 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 623968, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:52:44,322 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:44,322 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:44,322 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:44,322 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:44,322 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:44,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 829ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1285746, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:44,830 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630092_1099518554722] hdfs.DFSClient: Slow ReadProcessor read fields took 874ms (threshold=20ms); ack: seqno: 19083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 998372, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:45,074 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630091_1099518554721] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 19975 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 513539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:52:45,075 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:45,075 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:45,075 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:45,075 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:45,075 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:45,295 DEBUG [PriorityRpcServer.handler=19,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:52:45,296 WARN [MemStoreFlusher.0] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:52:45,296 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:52:45,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 832ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1332755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:45,830 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630092_1099518554722] hdfs.DFSClient: Slow ReadProcessor read fields took 872ms (threshold=20ms); ack: seqno: 19250 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1116447, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:45,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630092_1099518554722] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 19257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945258, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:45,901 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630092_1099518554722] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 19276 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736815, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:45,902 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630091_1099518554721] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 24829 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 528716, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:52:45,902 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:52:45,902 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:45,902 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:45,902 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:45,902 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:52:45,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 143ms (threshold=20ms); ack: seqno: 2919 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7737836, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:46,762 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630091_1099518554721] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 29689 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 605298, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:52:46,762 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:46,763 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:46,763 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:46,763 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:46,763 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:46,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 781ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1386059, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:46,830 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630092_1099518554722] hdfs.DFSClient: Slow ReadProcessor read fields took 824ms (threshold=20ms); ack: seqno: 19417 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196630, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:47,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630091_1099518554721] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 33864 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 495473, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:52:47,526 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:47,526 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:47,526 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:47,526 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:47,526 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:47,764 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 843ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1273114, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:47,830 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630092_1099518554722] hdfs.DFSClient: Slow ReadProcessor read fields took 881ms (threshold=20ms); ack: seqno: 19583 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1298696, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:48,306 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630091_1099518554721] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 38139 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 573833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:52:48,306 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:48,306 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:48,306 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:48,306 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:48,306 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:48,494 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:52:48,530 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025160988 with entries=77871, filesize=122.07 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025168494 2015-10-16 12:52:48,753 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630093_1099518554723] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 555 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 806993, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:48,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 843ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1649189, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:48,830 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630092_1099518554722] hdfs.DFSClient: Slow ReadProcessor read fields took 874ms (threshold=20ms); ack: seqno: 19750 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1420125, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:49,429 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630093_1099518554723] hdfs.DFSClient: Slow ReadProcessor read fields took 676ms (threshold=20ms); ack: seqno: 558 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 675880335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:49,430 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 676ms (threshold=20ms) 2015-10-16 12:52:49,430 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 676ms (threshold=20ms) 2015-10-16 12:52:49,430 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 675ms (threshold=20ms) 2015-10-16 12:52:49,430 INFO [sync.2] wal.FSHLog: Slow sync cost: 676 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:49,430 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 676ms (threshold=20ms) 2015-10-16 12:52:49,430 INFO [sync.0] wal.FSHLog: Slow sync cost: 676 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:49,430 INFO [sync.3] wal.FSHLog: Slow sync cost: 675 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:49,430 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 676ms (threshold=20ms) 2015-10-16 12:52:49,430 INFO [sync.4] wal.FSHLog: Slow sync cost: 677 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:49,430 INFO [sync.1] wal.FSHLog: Slow sync cost: 676 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:49,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 665ms (threshold=20ms); ack: seqno: 3345 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 666166159, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:49,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 272ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1397926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:49,843 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:49,843 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630093_1099518554723] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 2712 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 540358, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:49,843 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 3564 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 878327, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:49,843 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:49,845 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630092_1099518554722] hdfs.DFSClient: Slow ReadProcessor read fields took 891ms (threshold=20ms); ack: seqno: 19919 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1361492, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:50,635 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630093_1099518554723] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 7126 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 483512, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:50,635 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:50,635 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:50,635 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:52:50,635 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:50,635 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:52:50,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 834ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1470788, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:50,845 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630092_1099518554722] hdfs.DFSClient: Slow ReadProcessor read fields took 877ms (threshold=20ms); ack: seqno: 20086 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102675, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:51,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630093_1099518554723] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 11349 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 478511, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:51,407 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:51,407 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:51,408 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:51,408 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:51,407 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:51,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 858ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1636446, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:51,845 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630092_1099518554722] hdfs.DFSClient: Slow ReadProcessor read fields took 881ms (threshold=20ms); ack: seqno: 20253 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1322153, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:52,197 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630093_1099518554723] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 15584 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 612317, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:52,197 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:52:52,197 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:52,197 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:52,197 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:52,197 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:52,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630089_1099518554719] hdfs.DFSClient: Slow ReadProcessor read fields took 875ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1596718, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:52:52,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630092_1099518554722] hdfs.DFSClient: Slow ReadProcessor read fields took 882ms (threshold=20ms); ack: seqno: 20420 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1378352, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:53,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630093_1099518554723] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 19846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 595318, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:53,025 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:53,025 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:53,025 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:53,025 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:53,025 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:53,764 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 874ms (threshold=20ms); ack: seqno: 4181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1316813, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:53,847 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630092_1099518554722] hdfs.DFSClient: Slow ReadProcessor read fields took 884ms (threshold=20ms); ack: seqno: 20587 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1182533, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:53,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630093_1099518554723] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 24581 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 476932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:53,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 4333 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1016941, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:53,917 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:52:53,917 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630092_1099518554722] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 20630 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956765, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:53,917 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:52:53,917 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:52:54,764 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 835ms (threshold=20ms); ack: seqno: 4348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174206, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:54,839 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 4406 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 740258, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:54,847 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630092_1099518554722] hdfs.DFSClient: Slow ReadProcessor read fields took 842ms (threshold=20ms); ack: seqno: 20754 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1128284, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:54,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630093_1099518554723] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 29415 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 704828, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:55,710 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:52:55,710 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:52:55,710 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:52:55,710 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:52:55,710 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:52:55,711 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630093_1099518554723] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 33854 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 775477, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:55,764 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 852ms (threshold=20ms); ack: seqno: 4517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1243914, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:55,848 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 879ms (threshold=20ms); ack: seqno: 20924 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1646207, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:56,475 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630093_1099518554723] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 38020 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 527285, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:52:56,475 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:52:56,475 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:52:56,475 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:52:56,475 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:52:56,475 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:52:56,709 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:52:56,741 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:52:56,763 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025168494 with entries=77990, filesize=122.26 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025176710 2015-10-16 12:52:56,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 870ms (threshold=20ms); ack: seqno: 4684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1193193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:56,767 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947213, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:56,846 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#28 3999566/2798800 (142.90%), rate=21257.62 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=20.00 MB/sec, activeCompactions=2] 2015-10-16 12:52:56,857 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 880ms (threshold=20ms); ack: seqno: 21090 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 10926388, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:56,973 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#28 sleep 873 ms because current throughput is 78.74 MB/sec, max allowed is 10.00 MB/sec, already slept 18 time(s) and total slept time is 15488 ms till now. 2015-10-16 12:52:57,277 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 2661 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 453373, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:57,277 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:52:57,277 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:52:57,277 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:52:57,277 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:52:57,277 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:52:57,764 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 870ms (threshold=20ms); ack: seqno: 4851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1375066, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:57,848 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 875ms (threshold=20ms); ack: seqno: 21257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1150572, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:57,983 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 6340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 739847, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:58,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 6945 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 536004, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:58,119 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:52:58,119 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:52:58,119 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:52:58,119 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:52:58,119 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:52:58,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 860ms (threshold=20ms); ack: seqno: 5018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1219083, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:58,849 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 883ms (threshold=20ms); ack: seqno: 21424 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2351894, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:58,952 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 21489 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 809248, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:58,952 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 11608 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 472061, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:52:58,952 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:52:58,952 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:52:58,952 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:52:58,952 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:52:59,183 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 228ms (threshold=20ms); ack: seqno: 21495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1264828, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:59,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 868ms (threshold=20ms); ack: seqno: 5185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1353271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:59,848 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 592ms (threshold=20ms); ack: seqno: 21591 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1400718, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:52:59,896 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 21598 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1108409, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:00,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 391ms (threshold=20ms); ack: seqno: 21628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1650188, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:00,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 5352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1624051, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:01,117 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 812ms (threshold=20ms); ack: seqno: 21629 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812865349, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:01,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 5521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1754633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:01,865 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 5621 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 776579, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:02,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 905ms (threshold=20ms); ack: seqno: 21927 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 9628751, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:02,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 21933 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1206110, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:02,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 857ms (threshold=20ms); ack: seqno: 5688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1771487, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:03,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 851ms (threshold=20ms); ack: seqno: 22094 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1525548, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:03,251 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 22098 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1684588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:03,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 896ms (threshold=20ms); ack: seqno: 5855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1677023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:04,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 660ms (threshold=20ms); ack: seqno: 22263 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1531589, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:04,367 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 154ms (threshold=20ms); ack: seqno: 22268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1331647, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:04,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 6022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1409955, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:05,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 725ms (threshold=20ms); ack: seqno: 22430 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1535803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:05,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630094_1099518554724] hdfs.DFSClient: Slow ReadProcessor read fields took 888ms (threshold=20ms); ack: seqno: 6189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1868495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:05,911 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 704ms (threshold=20ms); ack: seqno: 22431 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2731992, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:05,969 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 22542 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 654092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:06,207 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 199ms (threshold=20ms); ack: seqno: 22597 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:06,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630097_1099518554727] hdfs.DFSClient: Slow ReadProcessor read fields took 891ms (threshold=20ms); ack: seqno: 6358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1625852, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:07,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630095_1099518554725] hdfs.DFSClient: Slow ReadProcessor read fields took 903ms (threshold=20ms); ack: seqno: 22764 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 9608031, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:07,764 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630097_1099518554727] hdfs.DFSClient: Slow ReadProcessor read fields took 897ms (threshold=20ms); ack: seqno: 6525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1453149, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:08,207 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 877ms (threshold=20ms); ack: seqno: 22933 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1379019, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:08,233 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 22940 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1145259, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:08,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630097_1099518554727] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 6694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1630448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:09,207 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 751ms (threshold=20ms); ack: seqno: 23100 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1408601, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:09,231 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 23107 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1144971, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:09,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630097_1099518554727] hdfs.DFSClient: Slow ReadProcessor read fields took 897ms (threshold=20ms); ack: seqno: 6861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1587457, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:09,799 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#30 average throughput is 12.65 MB/sec, slept 31 time(s) and total slept time is 26030 ms. 1 active compactions remaining, total limit is 20.00 MB/sec 2015-10-16 12:53:09,819 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8746f5b8d35444c6b01a74e88862866e as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8746f5b8d35444c6b01a74e88862866e 2015-10-16 12:53:09,833 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 10582ms (threshold=20ms); ack: seqno: 13424 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779841, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:09,833 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:53:09,837 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:53:09,844 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e011799159764a9683daebab3bc6f669, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e011799159764a9683daebab3bc6f669 2015-10-16 12:53:09,853 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2529360c9663442e921e83a9c263954e, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2529360c9663442e921e83a9c263954e 2015-10-16 12:53:09,861 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d4ecafb5ab5547b5b1f99629cfb4b736, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d4ecafb5ab5547b5b1f99629cfb4b736 2015-10-16 12:53:09,870 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/175cce2eb47d419f925f8f6d13169ccd, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/175cce2eb47d419f925f8f6d13169ccd 2015-10-16 12:53:09,870 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 8746f5b8d35444c6b01a74e88862866e(size=425.2 M), total size for store is 9.1 G. This selection was in queue for 0sec, and took 32sec to execute. 2015-10-16 12:53:09,870 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=425.2 M, priority=0, time=36903065949576694; duration=32sec 2015-10-16 12:53:09,870 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:53:09,870 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 6 compacting, 2 eligible, 10 blocking 2015-10-16 12:53:09,871 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:53:09,871 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:53:09,871 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:53:10,207 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 850ms (threshold=20ms); ack: seqno: 23267 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1538511, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:10,239 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 23272 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1352691, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:10,344 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 23322 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838492, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:10,500 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536923136 > lower bound=16777216 2015-10-16 12:53:10,500 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 12:53:10,602 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 769ms (threshold=20ms); ack: seqno: 13425 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 709602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:10,707 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 275ms (threshold=20ms); ack: seqno: 23434 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1366379, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:11,207 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 374ms (threshold=20ms); ack: seqno: 23601 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1186335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:11,256 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 23606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1335726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:11,368 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 23714 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1990667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:11,423 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 23721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866851, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:11,460 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 23725 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844230, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:11,527 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 23732 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 900428, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:11,568 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 23737 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 810426, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:11,628 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 23744 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021548, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:11,707 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 23768 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932230, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:11,768 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 23776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 777373, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:11,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 23812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 755338, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:11,985 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 23845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1258223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:12,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 164ms (threshold=20ms); ack: seqno: 23935 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870829, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:12,268 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 23944 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841167, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:12,709 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 338ms (threshold=20ms); ack: seqno: 24102 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1306572, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:12,802 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 24111 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1125274, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:13,045 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 24268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081867, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:13,209 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 163ms (threshold=20ms); ack: seqno: 24269 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910846, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:13,353 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=6291976, memsize=512.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/92b4ab0e585e4c4e814c0e61dc4e331c 2015-10-16 12:53:13,359 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/92b4ab0e585e4c4e814c0e61dc4e331c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/92b4ab0e585e4c4e814c0e61dc4e331c 2015-10-16 12:53:13,373 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/92b4ab0e585e4c4e814c0e61dc4e331c, entries=1864370, sequenceid=6291976, filesize=283.2 M 2015-10-16 12:53:13,409 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 2807ms (threshold=20ms); ack: seqno: 13426 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 35252394, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:13,409 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:53:13,409 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536922720, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2909ms, sequenceid=6291976, compaction requested=true 2015-10-16 12:53:13,409 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:53:13,410 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 6 compacting, 3 eligible, 10 blocking 2015-10-16 12:53:13,412 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:53:13,412 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:53:13,412 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:53:13,710 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 396ms (threshold=20ms); ack: seqno: 24440 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1531374, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:13,736 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 24445 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1097235, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:14,210 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 367ms (threshold=20ms); ack: seqno: 24607 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1499733, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:14,711 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 395ms (threshold=20ms); ack: seqno: 24774 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1470920, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:14,740 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 24783 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 778374, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:15,211 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630098_1099518554728] hdfs.DFSClient: Slow ReadProcessor read fields took 362ms (threshold=20ms); ack: seqno: 24941 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1416651, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:15,711 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 376ms (threshold=20ms); ack: seqno: 25110 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1376361, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:15,769 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 25116 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1219376, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:15,900 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 25264 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813469, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:15,928 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 25269 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868022, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:16,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 278ms (threshold=20ms); ack: seqno: 25277 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1406728, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:16,268 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 25282 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1284963, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:16,351 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 25319 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844877, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:16,711 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 281ms (threshold=20ms); ack: seqno: 25444 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1222812, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:16,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 25449 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1250632, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:16,910 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 25607 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797756, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:17,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 299ms (threshold=20ms); ack: seqno: 25611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1461097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:17,254 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 25619 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1150619, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:17,344 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 25627 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905597, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:17,400 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 25635 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 991171, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:17,419 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 4009ms (threshold=20ms); ack: seqno: 13427 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 699598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:17,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 25640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 899975, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:17,495 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 13443 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 669393, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:17,522 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 25652 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866507, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:17,558 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 13444 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 678305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:17,621 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 13446 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 682433, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:17,664 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 25742 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924034, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:17,711 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 25778 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 831843, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:17,777 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 25786 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:17,811 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 190ms (threshold=20ms); ack: seqno: 13448 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 882405, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:18,004 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 191ms (threshold=20ms); ack: seqno: 13458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 699912, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:18,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 334ms (threshold=20ms); ack: seqno: 25945 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1601101, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:18,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 25951 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1471189, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:18,579 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 574ms (threshold=20ms); ack: seqno: 13463 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769379, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:18,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 338ms (threshold=20ms); ack: seqno: 26112 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1613567, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:18,774 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 194ms (threshold=20ms); ack: seqno: 13464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 686516, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:18,842 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 26251 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069906, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:18,964 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 191ms (threshold=20ms); ack: seqno: 13465 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842329, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:19,140 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 175ms (threshold=20ms); ack: seqno: 13474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 810710, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:19,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 353ms (threshold=20ms); ack: seqno: 26279 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1679724, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:19,249 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 13475 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767464, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:19,279 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 26287 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1499144, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:19,359 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 13476 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847098, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:19,552 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 191ms (threshold=20ms); ack: seqno: 13481 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 679152, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:19,713 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 335ms (threshold=20ms); ack: seqno: 26446 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1334152, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:19,746 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 190ms (threshold=20ms); ack: seqno: 13501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 802453, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:19,843 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 26602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 815084, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:19,935 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 189ms (threshold=20ms); ack: seqno: 13505 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865903, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:20,123 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 188ms (threshold=20ms); ack: seqno: 13508 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905967, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:20,187 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 13513 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 846283, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:20,213 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 363ms (threshold=20ms); ack: seqno: 26615 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1433530, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:20,254 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 13514 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 746004, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:20,262 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 26620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1315389, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:20,340 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 13533 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 789139, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:20,365 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 26627 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 99416727, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:20,429 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 13540 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797427, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:20,517 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 13549 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 880503, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:20,584 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 13562 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 818500, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:20,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 299ms (threshold=20ms); ack: seqno: 26782 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1288528, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:21,213 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 360ms (threshold=20ms); ack: seqno: 26949 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1636524, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:21,308 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630102_1099518554732] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 27020 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 805510, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:21,395 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 17454 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 828275, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:21,713 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 320ms (threshold=20ms); ack: seqno: 27121 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1474904, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:22,199 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 21887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 557756, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:22,199 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:22,199 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:22,199 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:22,199 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:22,199 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:22,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 372ms (threshold=20ms); ack: seqno: 27287 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1278202, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:22,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 367ms (threshold=20ms); ack: seqno: 27454 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1458842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:22,847 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 27460 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1254189, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:22,986 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 26189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 563846, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:22,986 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 27616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 968272, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:22,986 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:22,986 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:22,986 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:22,986 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:23,213 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 223ms (threshold=20ms); ack: seqno: 27621 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1419756, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:23,368 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 27773 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934735, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:23,711 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 333ms (threshold=20ms); ack: seqno: 27788 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1090422, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:23,827 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 30460 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 398148, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:23,827 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:23,827 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:23,828 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:23,827 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 27904 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021068, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:23,828 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:23,827 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:23,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 27951 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1215936, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:24,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 247ms (threshold=20ms); ack: seqno: 27955 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1322174, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:24,689 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:24,689 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:53:24,689 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:24,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 35114 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 508166, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:24,689 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:24,690 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:24,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 382ms (threshold=20ms); ack: seqno: 28122 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1303933, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:24,997 DEBUG [PriorityRpcServer.handler=9,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:53:24,997 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134274608 > lower bound=16777216 2015-10-16 12:53:24,997 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:53:25,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 36809 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 633841, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:25,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 361ms (threshold=20ms); ack: seqno: 28289 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1041066, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:25,412 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 55ms (threshold=20ms) 2015-10-16 12:53:25,412 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 28431 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:25,412 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 55ms (threshold=20ms) 2015-10-16 12:53:25,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630096_1099518554726] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 38532 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 759954, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:25,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630104_1099518554734] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 433 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982487, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:53:25,526 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:53:25,578 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025176710 with entries=77911, filesize=122.13 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025205526 2015-10-16 12:53:25,579 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025152169 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025152169 2015-10-16 12:53:25,581 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630105_1099518554735] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1046545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:53:25,586 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025160988 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025160988 2015-10-16 12:53:25,594 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025168494 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025168494 2015-10-16 12:53:25,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 262ms (threshold=20ms); ack: seqno: 28456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1231854, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:25,919 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=6338608, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c84e8e74110a411ea1c87460326d3eeb 2015-10-16 12:53:25,924 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c84e8e74110a411ea1c87460326d3eeb as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c84e8e74110a411ea1c87460326d3eeb 2015-10-16 12:53:25,930 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c84e8e74110a411ea1c87460326d3eeb, entries=466280, sequenceid=6338608, filesize=70.8 M 2015-10-16 12:53:25,932 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134284560, currentsize=23.41 MB/24542640 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 935ms, sequenceid=6338608, compaction requested=true 2015-10-16 12:53:25,932 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:53:25,932 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 6 compacting, 4 eligible, 10 blocking 2015-10-16 12:53:25,934 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 891384066 starting at candidate #0 after considering 3 permutations with 1 in ratio 2015-10-16 12:53:25,934 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:53:25,935 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:53:25,935 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=850.1 M 2015-10-16 12:53:25,935 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8746f5b8d35444c6b01a74e88862866e, keycount=2798800, bloomtype=ROW, size=425.2 M, encoding=NONE, seqNum=6058900 2015-10-16 12:53:25,935 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bb87dca51ce74d4e8934761f39c8bcec, keycount=466300, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=6105534 2015-10-16 12:53:25,935 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/92b4ab0e585e4c4e814c0e61dc4e331c, keycount=1864370, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=6291976 2015-10-16 12:53:25,935 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c84e8e74110a411ea1c87460326d3eeb, keycount=466280, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=6338608 2015-10-16 12:53:25,955 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:53:26,055 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630105_1099518554735] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 2172 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 565523, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:53:26,055 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630106_1099518554736] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 82 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 685701, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:26,055 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:53:26,055 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:53:26,055 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:53:26,055 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:53:26,055 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:53:26,213 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 339ms (threshold=20ms); ack: seqno: 28623 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089772, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:26,550 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630105_1099518554735] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 4126 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 687993, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:53:26,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630106_1099518554736] hdfs.DFSClient: Slow ReadProcessor read fields took 840ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1367356, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:27,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630106_1099518554736] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 188 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 645358, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:27,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630105_1099518554735] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 6531 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 502409, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:53:27,012 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:27,012 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:27,012 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:27,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 868ms (threshold=20ms); ack: seqno: 28792 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1311110, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:27,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 141ms (threshold=20ms); ack: seqno: 28796 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1326690, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:27,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 28943 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941628, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:27,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630105_1099518554735] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 10849 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 680085, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:53:27,831 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:27,832 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:27,832 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:27,832 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:27,832 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:27,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630106_1099518554736] hdfs.DFSClient: Slow ReadProcessor read fields took 827ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1485894, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:28,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 690ms (threshold=20ms); ack: seqno: 28958 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1285173, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:28,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 28967 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989237, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:28,628 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630105_1099518554735] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 14992 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 395907, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:53:28,628 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:28,628 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:28,628 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:28,628 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:28,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630106_1099518554736] hdfs.DFSClient: Slow ReadProcessor read fields took 855ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1181400, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:29,213 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630103_1099518554733] hdfs.DFSClient: Slow ReadProcessor read fields took 851ms (threshold=20ms); ack: seqno: 29125 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1246422, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:29,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630105_1099518554735] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 19244 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 610928, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:53:29,459 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:53:29,459 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:53:29,459 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:53:29,459 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:29,459 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:29,724 DEBUG [PriorityRpcServer.handler=8,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:53:29,724 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134277168 > lower bound=16777216 2015-10-16 12:53:29,724 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:53:29,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630106_1099518554736] hdfs.DFSClient: Slow ReadProcessor read fields took 882ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1205285, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:30,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630108_1099518554738] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 530 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728417, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:30,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630105_1099518554735] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 23063 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 525643, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:53:30,212 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:30,212 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:30,212 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:30,212 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:30,212 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:30,214 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 873ms (threshold=20ms); ack: seqno: 29295 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1371829, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:30,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=6385241, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ae4112bd517046cfbd47fbd7accb0055 2015-10-16 12:53:30,624 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ae4112bd517046cfbd47fbd7accb0055 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ae4112bd517046cfbd47fbd7accb0055 2015-10-16 12:53:30,630 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ae4112bd517046cfbd47fbd7accb0055, entries=466290, sequenceid=6385241, filesize=70.8 M 2015-10-16 12:53:30,632 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134287120, currentsize=24.92 MB/26126400 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 908ms, sequenceid=6385241, compaction requested=false 2015-10-16 12:53:30,921 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630105_1099518554735] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 26750 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 422834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:53:30,922 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:53:30,922 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:53:30,922 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:53:30,922 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:53:30,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630106_1099518554736] hdfs.DFSClient: Slow ReadProcessor read fields took 883ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1131067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:31,214 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 866ms (threshold=20ms); ack: seqno: 29462 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1307738, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:31,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 29469 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1283157, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:31,478 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630105_1099518554735] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 28947 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 798820, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:53:31,885 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630105_1099518554735] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 30910 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 533719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:53:31,885 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:53:31,885 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:53:31,885 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:53:31,885 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:53:31,886 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:53:31,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630106_1099518554736] hdfs.DFSClient: Slow ReadProcessor read fields took 881ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1139676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:32,214 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 830ms (threshold=20ms); ack: seqno: 29629 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1441003, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:32,377 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 154ms (threshold=20ms); ack: seqno: 29638 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1320482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:32,539 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 29777 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1030414, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:32,669 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630105_1099518554735] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 35104 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 558504, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:53:32,669 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:53:32,669 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:32,669 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:53:32,669 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:53:32,669 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:32,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630106_1099518554736] hdfs.DFSClient: Slow ReadProcessor read fields took 884ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1276221, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:33,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 651ms (threshold=20ms); ack: seqno: 29798 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1403973, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:33,340 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:53:33,377 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025205526 with entries=77896, filesize=122.11 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025213340 2015-10-16 12:53:33,380 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102531, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:33,381 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025176710 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025176710 2015-10-16 12:53:33,479 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 400 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 513509, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:33,480 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:33,480 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:33,480 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:33,480 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:33,480 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:33,964 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630106_1099518554736] hdfs.DFSClient: Slow ReadProcessor read fields took 882ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1201130, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:34,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 862ms (threshold=20ms); ack: seqno: 29965 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1466300, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:34,327 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 30074 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 817307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:34,327 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 4943 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 549853, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:34,327 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:53:34,327 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:53:34,327 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:53:34,327 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:53:34,327 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:53:34,382 DEBUG [PriorityRpcServer.handler=2,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:53:34,382 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:53:34,382 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:53:34,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630106_1099518554736] hdfs.DFSClient: Slow ReadProcessor read fields took 876ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1226266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:35,187 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 9586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729610, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:35,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 846ms (threshold=20ms); ack: seqno: 30132 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1485944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:35,217 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 12:53:35,217 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 12:53:35,217 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 12:53:35,218 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 12:53:35,218 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 12:53:35,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630106_1099518554736] hdfs.DFSClient: Slow ReadProcessor read fields took 877ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1202705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:36,062 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630106_1099518554736] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 1772 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797891, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:36,099 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 13991 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 637904, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:36,202 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 14445 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 23000130, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:36,202 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:53:36,202 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:53:36,202 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:53:36,202 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:53:36,202 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:53:36,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 873ms (threshold=20ms); ack: seqno: 30298 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1402613, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:36,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 30304 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1134362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:36,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 18617 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 468390, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:36,958 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:36,958 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:36,958 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:36,958 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:36,958 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:36,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630106_1099518554736] hdfs.DFSClient: Slow ReadProcessor read fields took 863ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1378327, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:37,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 826ms (threshold=20ms); ack: seqno: 30465 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1372559, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:37,270 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 30474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1269227, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:37,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 22773 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 447009, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:37,735 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:53:37,735 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:53:37,735 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:53:37,735 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:53:37,735 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:53:37,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630106_1099518554736] hdfs.DFSClient: Slow ReadProcessor read fields took 879ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1202229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:38,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 812ms (threshold=20ms); ack: seqno: 30632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1550513, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:38,354 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 30637 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1300934, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:38,499 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 26429 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 143614420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:38,499 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 144ms (threshold=20ms) 2015-10-16 12:53:38,499 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 144ms (threshold=20ms) 2015-10-16 12:53:38,499 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 144ms (threshold=20ms) 2015-10-16 12:53:38,499 INFO [sync.4] wal.FSHLog: Slow sync cost: 143 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:38,499 INFO [sync.0] wal.FSHLog: Slow sync cost: 143 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:38,499 INFO [sync.2] wal.FSHLog: Slow sync cost: 143 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:38,499 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 144ms (threshold=20ms) 2015-10-16 12:53:38,500 INFO [sync.1] wal.FSHLog: Slow sync cost: 144 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:38,499 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 144ms (threshold=20ms) 2015-10-16 12:53:38,500 INFO [sync.3] wal.FSHLog: Slow sync cost: 144 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:38,505 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 30777 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 990195, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:38,632 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 26981 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 496932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:38,632 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:53:38,632 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:53:38,632 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:53:38,632 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:53:38,632 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:53:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 12:53:38,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630110_1099518554740] hdfs.DFSClient: Slow ReadProcessor read fields took 875ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1614239, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:39,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 696ms (threshold=20ms); ack: seqno: 30799 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1412889, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:39,255 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 30800 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 39074985, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:39,309 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630110_1099518554740] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 2584 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5555657, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:39,309 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630107_1099518554737] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 30846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7371142, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:39,309 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 30592 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 452445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:39,309 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:39,309 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:39,310 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:39,310 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:39,310 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:39,693 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 32618 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 532777, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:39,693 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630111_1099518554741] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 31281 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3532736, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:39,693 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:39,693 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:39,693 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:39,693 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630110_1099518554740] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 3033 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 884410, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:39,693 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:39,693 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:40,071 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630110_1099518554740] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 3469 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 676686, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:40,071 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 34663 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 514230, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:40,071 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630111_1099518554741] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 31714 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 916857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:40,071 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:40,072 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:40,072 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:40,072 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:40,071 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:40,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630111_1099518554741] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 31802 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948952, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:40,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630111_1099518554741] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 32118 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 731361, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:40,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 36739 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 474975, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:40,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630110_1099518554740] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 3923 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 753282, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:40,464 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:53:40,464 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:53:40,464 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:53:40,464 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:40,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 37004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 824210, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:40,889 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630111_1099518554741] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 32600 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 775481, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:40,890 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630109_1099518554739] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 37997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 663880, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:53:40,890 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630112_1099518554742] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 4485 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956333, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:53:41,144 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:53:41,185 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025213340 with entries=77886, filesize=122.10 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025221144 2015-10-16 12:53:41,188 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985781, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:41,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 422 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 533882, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:41,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630112_1099518554742] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 4936 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 773115, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:53:41,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630111_1099518554741] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 33037 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 726520, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:41,300 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:53:41,300 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:53:41,300 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:53:41,300 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:53:41,301 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:53:41,695 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 2307 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 515739, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:41,695 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630112_1099518554742] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 5435 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767990, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:53:41,695 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:53:41,695 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:53:41,695 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:53:41,695 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:53:41,695 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:53:41,696 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630114_1099518554744] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 33438 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950828, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:42,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630114_1099518554744] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 33841 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834844, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:42,098 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:42,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630112_1099518554742] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 5901 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728877, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:53:42,098 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:42,098 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:42,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 4249 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 445264, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:42,098 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:42,098 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:42,526 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:42,526 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:42,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 6404 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 501197, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:42,527 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:42,527 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:42,528 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630114_1099518554744] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 34245 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1049009, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:42,636 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 6887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 20985335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:42,636 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:53:42,636 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:53:42,636 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:53:42,636 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:53:42,636 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:53:42,931 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 8340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 614612, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:42,931 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630114_1099518554744] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 34629 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 783372, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:42,931 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630115_1099518554745] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 6826 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 785601, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:42,931 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:42,931 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:43,309 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:43,309 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:43,309 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630115_1099518554745] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 7316 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870477, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:43,309 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630114_1099518554744] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 35038 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767694, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:43,309 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:43,310 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 10180 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 627139, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:43,680 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630116_1099518554746] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 35468 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967269, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:43,680 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 12040 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 814535, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:43,680 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:43,680 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:43,680 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:43,680 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:43,680 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:43,681 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630115_1099518554745] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 7797 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 876539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:53:44,073 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630116_1099518554746] hdfs.DFSClient: Slow ReadProcessor read fields took 154ms (threshold=20ms); ack: seqno: 35694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 6602033, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:44,117 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630116_1099518554746] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 35716 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1028750, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:44,117 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 14128 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 530967, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:44,117 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:44,117 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630117_1099518554747] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 8389 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:53:44,118 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:44,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630117_1099518554747] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 8924 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 832750, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:53:44,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 15921 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 534588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:44,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630116_1099518554746] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 36112 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 858468, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:44,526 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:53:44,526 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:53:44,526 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:53:44,526 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:53:44,526 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:53:44,935 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630117_1099518554747] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 9426 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861841, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:53:44,935 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 17675 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 610737, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:44,936 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:44,936 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:44,936 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:44,936 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:44,936 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:44,939 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630116_1099518554746] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 36535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 805673, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:45,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630116_1099518554746] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 36589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 744002, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:45,348 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630117_1099518554747] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 9940 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 630011, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:53:45,349 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630116_1099518554746] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 36952 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 828517, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:45,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 134ms (threshold=20ms); ack: seqno: 19437 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 698388, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:45,645 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630116_1099518554746] hdfs.DFSClient: Slow ReadProcessor read fields took 191ms (threshold=20ms); ack: seqno: 37085 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936908, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:45,824 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630116_1099518554746] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 37196 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 978684, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:45,860 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 21680 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 600878, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:45,860 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:45,860 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:45,860 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:45,860 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:45,861 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:53:45,862 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630116_1099518554746] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 37197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 903658, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:45,910 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630116_1099518554746] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 37204 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1074866, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:46,062 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630117_1099518554747] hdfs.DFSClient: Slow ReadProcessor read fields took 373ms (threshold=20ms); ack: seqno: 10418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 951503, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:53:46,065 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630116_1099518554746] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 37327 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924617, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:46,361 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630119_1099518554749] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 37595 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905134, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:53:46,361 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 24208 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 495320, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:46,361 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:46,361 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:46,361 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:46,361 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:46,361 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630118_1099518554748] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 10857 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 756745, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:53:46,361 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:46,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 26057 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 627973, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:46,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630119_1099518554749] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 38010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762943, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:53:46,743 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630118_1099518554748] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 11333 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4162466, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:53:46,743 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:46,743 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:46,743 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:46,743 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:46,743 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:47,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630119_1099518554749] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 38432 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1105031, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:53:47,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630118_1099518554748] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 11810 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 829487, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:53:47,119 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:47,119 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:47,119 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:47,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 27951 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 613833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:47,120 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:47,120 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:47,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630119_1099518554749] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 38629 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 771873, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:53:47,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630119_1099518554749] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 38905 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885831, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:53:47,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630118_1099518554748] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 12295 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 815468, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:53:47,556 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 29624 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 809148, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:47,952 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 12:53:47,952 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 12:53:47,952 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630119_1099518554749] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 39324 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963923, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:53:47,953 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 31436 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 766134, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:47,954 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630120_1099518554750] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 12780 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1071604, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:48,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630119_1099518554749] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 39523 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1034404, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:53:48,382 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 39645 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 20371130, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:48,382 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 33420 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 546860, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:48,382 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630120_1099518554750] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 13305 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1138619, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:48,382 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:53:48,382 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:53:48,520 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 39666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1249546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:48,554 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 39672 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1262776, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:48,613 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 39678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1247816, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:48,769 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#31 average throughput is 36.24 MB/sec, slept 13 time(s) and total slept time is 11281 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 12:53:48,769 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 39813 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1331143, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:48,800 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a289bcb11d5f4574b597f51f09338490 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a289bcb11d5f4574b597f51f09338490 2015-10-16 12:53:48,815 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:53:48,820 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:53:48,825 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8746f5b8d35444c6b01a74e88862866e, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8746f5b8d35444c6b01a74e88862866e 2015-10-16 12:53:48,834 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bb87dca51ce74d4e8934761f39c8bcec, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bb87dca51ce74d4e8934761f39c8bcec 2015-10-16 12:53:48,842 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/92b4ab0e585e4c4e814c0e61dc4e331c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/92b4ab0e585e4c4e814c0e61dc4e331c 2015-10-16 12:53:48,850 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c84e8e74110a411ea1c87460326d3eeb, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c84e8e74110a411ea1c87460326d3eeb 2015-10-16 12:53:48,851 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into a289bcb11d5f4574b597f51f09338490(size=850.0 M), total size for store is 9.5 G. This selection was in queue for 0sec, and took 22sec to execute. 2015-10-16 12:53:48,851 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=850.1 M, priority=0, time=36903114788561991; duration=22sec 2015-10-16 12:53:48,851 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:53:48,851 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 6 compacting, 2 eligible, 10 blocking 2015-10-16 12:53:48,852 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:53:48,852 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:53:48,852 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:53:48,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 35782 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 563345, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:48,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 39925 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:48,887 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:53:48,887 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:53:48,887 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:53:48,887 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:53:48,887 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:53:48,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 39930 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174883, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:49,113 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 40067 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1191130, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:49,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 40189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948958, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:49,464 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 40325 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:49,561 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 40341 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101902, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:49,682 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536928256 > lower bound=16777216 2015-10-16 12:53:49,682 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 12:53:49,749 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 40472 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1225542, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:49,785 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 778ms (threshold=20ms); ack: seqno: 36433 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 800662, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:49,929 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 40616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:50,100 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 40773 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954973, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:50,153 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630122_1099518554752] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 584 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 760032, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:53:50,153 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 40807 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 705547, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:50,234 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 40816 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965536, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:50,406 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 40972 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1172227, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:50,592 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 41033 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1028806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:50,751 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 41183 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:50,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630121_1099518554751] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 41332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940631, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:53:51,622 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630123_1099518554753] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 3103 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762354, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:51,622 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630124_1099518554754] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 42349 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 832890, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:51,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630124_1099518554754] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 42679 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 814470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:52,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630123_1099518554753] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 4158 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 849268, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:52,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630124_1099518554754] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 43224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 944074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:52,509 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=6571685, memsize=512.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a87966246b014f83994b5da26116f5ea 2015-10-16 12:53:52,516 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a87966246b014f83994b5da26116f5ea as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a87966246b014f83994b5da26116f5ea 2015-10-16 12:53:52,529 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a87966246b014f83994b5da26116f5ea, entries=1864390, sequenceid=6571685, filesize=283.2 M 2015-10-16 12:53:52,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 2746ms (threshold=20ms); ack: seqno: 36434 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 734110, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:52,531 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536927840, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2849ms, sequenceid=6571685, compaction requested=true 2015-10-16 12:53:52,531 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:53:52,531 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 6 compacting, 3 eligible, 10 blocking 2015-10-16 12:53:52,533 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:53:52,533 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:53:52,533 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:53:53,012 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630126_1099518554756] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 44358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 682389, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:53:53,116 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 585ms (threshold=20ms); ack: seqno: 36435 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 694660, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:53,595 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630126_1099518554756] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 45132 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 791303, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:53:53,595 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630113_1099518554743] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 38759 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 513244, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:53:53,595 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:53:53,595 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:53:53,595 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:53:53,595 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:53:53,595 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:53:53,653 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:53:53,692 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025221144 with entries=77747, filesize=121.88 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025233653 2015-10-16 12:53:53,695 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025205526 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025205526 2015-10-16 12:53:53,696 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 961667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:53,701 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025213340 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025213340 2015-10-16 12:53:54,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 2320 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 478222, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:54,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630126_1099518554756] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 45821 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769764, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:53:54,133 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:54,133 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:54,134 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:54,134 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:54,134 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:54,683 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 5120 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 488764, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:54,683 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630128_1099518554758] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 46503 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864870, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:53:54,683 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:54,683 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:54,683 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:54,683 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:54,683 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:55,032 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630128_1099518554758] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 46830 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902793, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:53:55,057 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 6489 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741487, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:55,178 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630128_1099518554758] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 46993 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 843415, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:53:55,334 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 7936 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 477415, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:55,334 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:53:55,334 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:53:55,334 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:53:55,334 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:53:55,334 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630128_1099518554758] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 47159 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 683815, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:53:55,334 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:53:55,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 10922 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 491138, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:55,879 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:55,879 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:55,879 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:55,879 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:55,879 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:55,889 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630128_1099518554758] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 47791 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 871992, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:53:56,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 13739 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 544191, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:56,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630129_1099518554759] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 48456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 854718, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:56,413 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:56,413 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:56,413 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:56,413 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:56,413 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:56,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630129_1099518554759] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 48842 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 816297, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:56,846 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#28 13823900/10109452 (136.74%), rate=28452.48 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=unlimited, activeCompactions=1] 2015-10-16 12:53:56,965 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 16640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 501052, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:56,965 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630129_1099518554759] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 49116 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 789837, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:56,965 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:56,965 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:56,965 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:56,965 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:56,965 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:53:57,492 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:57,492 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:57,492 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630129_1099518554759] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 49766 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1080683, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:53:57,492 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 19467 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 460734, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:57,492 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:57,493 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:57,494 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:57,707 DEBUG [PriorityRpcServer.handler=16,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:53:57,707 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134274944 > lower bound=16777216 2015-10-16 12:53:57,707 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:53:57,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 20672 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 652552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:58,013 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630131_1099518554761] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 343 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 749632, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:58,013 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 21808 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 612765, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:58,013 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630130_1099518554760] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 50379 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 765021, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:58,013 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:53:58,013 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:53:58,013 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:53:58,013 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:53:58,013 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:53:58,382 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630130_1099518554760] hdfs.DFSClient: Slow ReadProcessor read fields took 140ms (threshold=20ms); ack: seqno: 50682 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:58,516 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630130_1099518554760] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 50823 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861460, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:58,516 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 24204 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 552365, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:58,516 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630131_1099518554761] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 1031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:53:58,516 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:53:58,516 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:53:58,516 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:53:58,516 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:53:58,516 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:53:58,608 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=6618314, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c65c8eb5eb134ea4917542070ea7f4ba 2015-10-16 12:53:58,614 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c65c8eb5eb134ea4917542070ea7f4ba as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c65c8eb5eb134ea4917542070ea7f4ba 2015-10-16 12:53:58,620 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c65c8eb5eb134ea4917542070ea7f4ba, entries=466250, sequenceid=6618314, filesize=70.8 M 2015-10-16 12:53:58,622 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134275680, currentsize=22.70 MB/23802400 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 915ms, sequenceid=6618314, compaction requested=true 2015-10-16 12:53:58,622 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:53:58,622 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 6 compacting, 4 eligible, 10 blocking 2015-10-16 12:53:58,624 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio 2015-10-16 12:53:58,624 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:53:58,624 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:53:58,683 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630130_1099518554760] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 50969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 791191, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:58,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630130_1099518554760] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 51112 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 855481, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:59,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630130_1099518554760] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 51235 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:59,122 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 27423 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 477425, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:59,122 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630130_1099518554760] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 51316 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 890087, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:59,122 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:59,122 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:59,122 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:59,122 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:59,123 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:53:59,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630130_1099518554760] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 51997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741796, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:53:59,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 30233 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 676661, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:53:59,659 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:59,659 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:53:59,659 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:53:59,852 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630130_1099518554760] hdfs.DFSClient: Slow ReadProcessor read fields took 150ms (threshold=20ms); ack: seqno: 52064 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023628, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:00,046 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 31671 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5056540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:54:00,113 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 52388 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1156067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:00,256 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 52522 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 872290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:00,356 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 52625 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926679, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:00,356 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 33266 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 640667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:54:00,357 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:54:00,357 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:54:00,357 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:54:00,357 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:54:00,357 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:54:00,416 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 52679 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:00,585 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 52820 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 878305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:00,757 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 52967 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:00,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 53075 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038727, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:00,996 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 53117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 719982, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:00,996 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 36883 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 557231, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:54:00,996 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:54:00,996 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:54:00,996 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:54:00,996 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:54:00,996 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 12:54:01,198 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 197ms (threshold=20ms); ack: seqno: 53125 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1156785, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:01,331 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:54:01,351 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 53253 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936421, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:01,516 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 53377 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 753211, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:01,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 53404 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 923613, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:01,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 40936 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 419330, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:54:01,712 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:01,712 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:01,712 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:01,712 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:01,713 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:01,970 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630127_1099518554757] hdfs.DFSClient: Slow ReadProcessor read fields took 249ms (threshold=20ms); ack: seqno: 40981 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2658336, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:54:01,976 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 634ms (threshold=20ms) 2015-10-16 12:54:02,016 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 296ms (threshold=20ms) 2015-10-16 12:54:02,016 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 295ms (threshold=20ms) 2015-10-16 12:54:02,016 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 296ms (threshold=20ms) 2015-10-16 12:54:02,016 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 296ms (threshold=20ms) 2015-10-16 12:54:02,016 INFO [sync.2] wal.FSHLog: Slow sync cost: 295 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:54:02,016 INFO [sync.0] wal.FSHLog: Slow sync cost: 296 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:54:02,016 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 296ms (threshold=20ms) 2015-10-16 12:54:02,016 INFO [sync.3] wal.FSHLog: Slow sync cost: 296 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:54:02,016 INFO [sync.1] wal.FSHLog: Slow sync cost: 296 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:54:02,016 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Slow sync cost: 674 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:54:02,034 INFO [sync.4] wal.FSHLog: Slow sync cost: 313 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:02,035 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025233653 with entries=81669, filesize=128.03 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025241331 2015-10-16 12:54:02,037 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025221144 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025221144 2015-10-16 12:54:02,038 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1134127, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:02,576 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 937ms (threshold=20ms); ack: seqno: 53422 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936352870, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:02,626 DEBUG [PriorityRpcServer.handler=14,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:54:02,626 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134265136 > lower bound=16777216 2015-10-16 12:54:02,627 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:54:02,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 53510 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 732845, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:02,656 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 3101 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 648851, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:02,863 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 53608 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994703, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:02,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 4235 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 706956, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:02,907 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:02,907 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:02,907 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630135_1099518554765] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 324 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915855, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:02,907 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:02,907 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 53619 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792073, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:02,907 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:02,907 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:03,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 6500 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 513578, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:03,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630132_1099518554762] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 54148 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 712429, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:03,355 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:54:03,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630135_1099518554765] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 938 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 737292, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:03,355 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:54:03,355 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:54:03,355 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:54:03,355 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:54:03,508 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=6664941, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/73de7b1a6d184505bea1cc92489215fb 2015-10-16 12:54:03,513 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/73de7b1a6d184505bea1cc92489215fb as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/73de7b1a6d184505bea1cc92489215fb 2015-10-16 12:54:03,519 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/73de7b1a6d184505bea1cc92489215fb, entries=466230, sequenceid=6664941, filesize=70.8 M 2015-10-16 12:54:03,522 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134270480, currentsize=23.17 MB/24300720 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 895ms, sequenceid=6664941, compaction requested=true 2015-10-16 12:54:03,522 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:54:03,522 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 6 compacting, 5 eligible, 10 blocking 2015-10-16 12:54:03,524 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 6 permutations with 0 in ratio 2015-10-16 12:54:03,524 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:54:03,524 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:54:03,866 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630136_1099518554766] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 54760 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 776051, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:03,866 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 9075 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 485002, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:03,866 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:54:03,866 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:54:03,866 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:54:03,867 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:54:03,867 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:54:04,401 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 11841 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 581542, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:04,401 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630136_1099518554766] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 55432 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728918, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:04,402 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:54:04,402 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:54:04,402 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:54:04,402 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:54:04,402 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 12:54:04,952 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 14425 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 405312, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:04,952 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630136_1099518554766] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 56151 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741208, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:04,952 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:04,952 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:04,952 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:04,952 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:04,952 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:05,148 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 14768 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:05,535 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630137_1099518554767] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 56960 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862816, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:05,535 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 16650 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672586, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:05,535 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:54:05,535 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:54:05,535 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:54:05,535 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:54:05,535 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:54:06,080 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 19264 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 571359, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:06,080 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630137_1099518554767] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 57695 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 673398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:06,080 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:06,080 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:06,080 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:06,080 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:06,080 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:06,299 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630137_1099518554767] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 57842 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 120430247, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:06,674 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 22002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 488757, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:06,674 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630138_1099518554768] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 58372 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2842941, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:06,674 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:54:06,675 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:54:07,205 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630138_1099518554768] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 58955 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834176, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:07,264 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:07,264 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:07,264 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:07,264 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:07,264 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630138_1099518554768] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 58998 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 699121, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:07,264 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:07,343 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630138_1099518554768] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 59036 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 730630, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:07,509 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:54:07,509 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:54:07,509 DEBUG [PriorityRpcServer.handler=4,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:54:07,509 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 6 compacting, 5 eligible, 10 blocking 2015-10-16 12:54:07,511 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 6 permutations with 0 in ratio 2015-10-16 12:54:07,511 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:54:07,511 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:54:07,569 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630138_1099518554768] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 59250 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 854029, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:07,687 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630138_1099518554768] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 59303 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834033, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:07,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630138_1099518554768] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 59459 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 832551, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:07,941 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 12:54:07,941 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 12:54:07,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 28489 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862486, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:07,943 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630138_1099518554768] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 59494 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 978265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:08,006 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630138_1099518554768] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 59503 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 786018, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:08,193 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630138_1099518554768] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 59635 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:08,371 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630138_1099518554768] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 59773 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 721763, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:08,538 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630138_1099518554768] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 59965 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 802283, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:08,606 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630138_1099518554768] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 60019 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 694673, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:08,606 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 31888 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 625926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:08,606 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:08,606 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:08,606 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:08,606 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:08,606 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:08,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630138_1099518554768] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 60190 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 878141, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:09,038 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630138_1099518554768] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 60359 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908503, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:09,179 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630139_1099518554769] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 60481 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013903, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:09,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630139_1099518554769] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 60564 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:09,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 35244 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 669731, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:09,285 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:54:09,285 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:54:09,285 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:54:09,285 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:54:09,285 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:54:09,386 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630139_1099518554769] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 60615 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941590, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:09,868 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 37734 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 829129, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:09,926 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630133_1099518554763] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 37874 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 510359, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:09,926 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630139_1099518554769] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 61280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 631220, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:09,926 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:09,926 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:10,119 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:54:10,209 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 84ms (threshold=20ms) 2015-10-16 12:54:10,225 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025241331 with entries=78581, filesize=123.18 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025250119 2015-10-16 12:54:10,226 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025233653 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025233653 2015-10-16 12:54:10,296 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630139_1099518554769] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 61690 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859401, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:10,517 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630140_1099518554770] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 1377 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 514264, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:54:10,517 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630139_1099518554769] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 61907 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 696121, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:10,517 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:10,517 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:10,517 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:10,518 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:10,518 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:10,579 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630139_1099518554769] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 61915 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869355, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:10,740 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630139_1099518554769] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 62054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 851661, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:10,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630139_1099518554769] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 62179 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911645, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:11,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630139_1099518554769] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 62315 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844526, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:11,170 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630140_1099518554770] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 4965 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 476225, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:54:11,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630139_1099518554769] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 62395 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 784034, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:11,171 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:11,171 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:11,171 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:11,171 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:11,171 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:11,229 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630139_1099518554769] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 62401 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 791601, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:54:11,411 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 62527 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1680286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:11,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 62660 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767410, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:11,773 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 62794 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845707, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:11,852 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630140_1099518554770] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 8546 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 592913, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:54:11,852 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 62862 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 724622, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:11,852 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:11,852 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:11,852 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:11,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 62865 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 875269, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:12,060 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630140_1099518554770] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 9211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 88511799, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:54:12,060 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 89ms (threshold=20ms) 2015-10-16 12:54:12,060 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 89ms (threshold=20ms) 2015-10-16 12:54:12,060 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 89ms (threshold=20ms) 2015-10-16 12:54:12,060 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 89ms (threshold=20ms) 2015-10-16 12:54:12,060 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 89ms (threshold=20ms) 2015-10-16 12:54:12,110 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 63014 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 878884, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:12,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 63129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:12,544 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 63256 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936791, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:12,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630140_1099518554770] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 12196 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489324, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:54:12,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 63326 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 722361, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:12,624 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:12,624 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:12,624 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:12,624 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:12,624 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:12,705 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 63334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920807, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:12,908 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 63466 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 931098, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:13,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 63473 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 900978, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:13,077 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 63481 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 665774, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:13,146 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 63488 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 686325, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:13,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 63616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865763, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:13,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630140_1099518554770] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 16349 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:54:13,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 63682 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 791574, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:13,399 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:13,399 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:54:13,399 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:54:13,399 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:13,399 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 12:54:13,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 63687 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 891994, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:13,600 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 63776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912647, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:13,777 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 63907 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:13,956 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 64041 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794171, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:14,090 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630140_1099518554770] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 19894 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 521526, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:54:14,090 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:54:14,090 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:54:14,090 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:54:14,090 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:54:14,090 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:54:14,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 64189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:14,309 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630140_1099518554770] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 20409 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 680106, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:54:14,329 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 64331 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902631, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:14,515 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630141_1099518554771] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 64470 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913993, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:14,695 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 64610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1381842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:14,833 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630140_1099518554770] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 23241 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 469252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:54:14,833 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:14,833 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:14,833 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:14,833 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:54:14,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 64732 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:15,095 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 64876 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 856848, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:15,353 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 154ms (threshold=20ms); ack: seqno: 65018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1009546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:15,532 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630140_1099518554770] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 27064 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 719773, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:54:15,532 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:54:15,532 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:54:15,532 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:54:15,532 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:54:15,532 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:54:15,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 65160 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861397, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:15,586 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 65168 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 951160, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:15,758 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 65311 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:15,911 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 65397 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011921, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:16,020 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 65445 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868310, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:16,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 65581 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909160, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:16,240 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630140_1099518554770] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 30681 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 571204, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:54:16,240 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 65637 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873490, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:16,240 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:54:16,240 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:54:16,240 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:54:16,240 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:54:16,240 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:54:16,724 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 66174 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950217, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:16,822 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630140_1099518554770] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 33673 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 613160, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:54:16,847 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:54:16,822 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 66261 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 691676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:16,847 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 60ms (threshold=20ms) 2015-10-16 12:54:16,847 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 61ms (threshold=20ms) 2015-10-16 12:54:16,847 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 60ms (threshold=20ms) 2015-10-16 12:54:16,847 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:54:16,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 66270 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2762218, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:17,033 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 66394 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1094632, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:17,161 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 66503 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841184, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:17,298 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630140_1099518554770] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 35643 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 556748, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:54:17,298 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630142_1099518554772] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 66585 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819942, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:17,298 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 80ms (threshold=20ms) 2015-10-16 12:54:17,299 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 80ms (threshold=20ms) 2015-10-16 12:54:17,298 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 80ms (threshold=20ms) 2015-10-16 12:54:17,299 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 79ms (threshold=20ms) 2015-10-16 12:54:17,299 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 80ms (threshold=20ms) 2015-10-16 12:54:17,511 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 66834 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845412, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:17,550 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630140_1099518554770] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 36826 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 578917, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:54:17,550 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:17,550 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 66846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 722104, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:17,550 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:17,550 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:17,550 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:17,550 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:54:17,980 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:54:18,033 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025250119 with entries=77939, filesize=122.18 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025257980 2015-10-16 12:54:18,036 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924344, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:18,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 67454 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 872355, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:18,153 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 67480 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736556, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:18,153 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 673178, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:18,153 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:54:18,153 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:54:18,153 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:54:18,153 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:54:18,153 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:54:18,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 67485 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 976165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:18,370 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 67615 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 987418, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:18,545 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 67742 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021049, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:18,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 2320 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 654726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:18,727 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 67862 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1003854, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:18,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 67915 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1100669, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:18,926 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 4179 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 469346, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:18,927 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:54:18,927 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:54:18,927 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:54:18,927 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:54:18,927 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:54:18,928 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 67920 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1077141, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:19,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 67924 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1029614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:19,196 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 68056 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888626, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:19,393 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 68169 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 977833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:19,574 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 68295 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942693, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:19,623 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:54:19,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 68312 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 755348, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:19,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 8223 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 542327, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:19,623 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:54:19,623 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:54:19,623 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:54:19,623 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:54:19,700 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 68319 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1090675, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:19,729 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 8761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 21045873, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:19,729 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:54:19,729 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:54:19,729 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:54:19,729 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:54:19,729 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:54:19,927 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 68438 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:20,082 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 68568 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927288, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:20,218 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 11235 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 584884, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:20,218 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 68ms (threshold=20ms) 2015-10-16 12:54:20,218 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 68ms (threshold=20ms) 2015-10-16 12:54:20,225 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:54:20,225 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:54:20,225 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:54:20,250 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 68705 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 929204, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:20,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 68716 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865412, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:20,359 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 68720 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:20,401 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 68723 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 837803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:20,401 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 12123 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 420312, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:20,401 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:54:20,401 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:54:20,401 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:54:20,401 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:54:20,490 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630143_1099518554773] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 68732 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954818, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:21,235 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:54:21,235 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:54:21,235 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:54:21,235 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:54:21,235 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:54:21,607 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 68858 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1199344, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:21,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 68993 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1155827, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:21,968 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 69146 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1134044, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:22,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 69191 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023319, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:22,289 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 69344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1041652, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:22,456 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 69495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1150280, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:22,637 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 69650 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1165281, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:22,819 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 69801 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1142168, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:22,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 69880 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 789248, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:23,000 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 69887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1202028, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:23,189 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 70038 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:23,389 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 70182 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1118579, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:23,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 70334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1073239, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:23,737 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 70418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1071856, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:23,907 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 70456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1152981, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:24,071 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 70605 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1039675, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:24,247 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630145_1099518554775] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 70763 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836563, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:24,409 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 70914 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1018378, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:24,704 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 71235 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 803051, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:25,751 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 633ms (threshold=20ms); ack: seqno: 71872 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1516498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:26,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 166ms (threshold=20ms); ack: seqno: 72010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:26,056 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 72015 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 918483, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:26,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 168ms (threshold=20ms); ack: seqno: 72023 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928322, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:26,392 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 160ms (threshold=20ms); ack: seqno: 72031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947018, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:26,452 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 72034 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014901, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:26,615 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 161ms (threshold=20ms); ack: seqno: 72039 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1027047, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:26,676 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 72045 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081383, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:26,976 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 300ms (threshold=20ms); ack: seqno: 72046 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1407267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:27,763 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 676ms (threshold=20ms); ack: seqno: 72225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1485144, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:27,805 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 72229 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:28,418 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 610ms (threshold=20ms); ack: seqno: 72235 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1639650, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:28,661 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 234ms (threshold=20ms); ack: seqno: 72245 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1408852, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:28,980 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 313ms (threshold=20ms); ack: seqno: 72256 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1257892, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:29,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 72537 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 901050, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:29,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 72572 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 708472, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:30,100 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630146_1099518554776] hdfs.DFSClient: Slow ReadProcessor read fields took 725ms (threshold=20ms); ack: seqno: 72706 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1413116, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:30,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 72967 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1475126, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:30,791 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 270ms (threshold=20ms); ack: seqno: 73126 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1520565, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:31,176 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 262ms (threshold=20ms); ack: seqno: 73301 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1250070, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:31,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 73307 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1423958, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:31,504 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 73611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921489, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:31,681 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 73762 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023992, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:31,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 73871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013178, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:31,925 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 73936 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 764486, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:31,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 73942 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969087, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:32,126 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=719, evicted=0, evictedPerRun=0.0 2015-10-16 12:54:32,176 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 74076 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011491, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:32,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 74210 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060450, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:32,539 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 74338 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 858915, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:32,724 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 74475 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982151, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:32,902 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 74611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969730, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:33,137 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 134ms (threshold=20ms); ack: seqno: 74746 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:33,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 74881 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 878322, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:33,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630147_1099518554777] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 75017 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:33,610 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630149_1099518554779] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 75116 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1254635, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:33,805 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630149_1099518554779] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 75275 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1158743, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:33,836 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630149_1099518554779] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 75280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986412, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:33,901 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630149_1099518554779] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 75291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1059102, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:34,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630149_1099518554779] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 75441 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038093, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:34,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630149_1099518554779] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 75598 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1133266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:34,403 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630149_1099518554779] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 75688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919684, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:34,559 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630149_1099518554779] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 75838 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1061833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:34,630 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630149_1099518554779] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 75846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:34,853 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630149_1099518554779] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 76008 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1108712, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:35,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630149_1099518554779] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 76625 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 810061, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:35,664 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630149_1099518554779] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 77005 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1053201, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:36,361 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630150_1099518554780] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 77976 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 704106, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:54:36,480 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630150_1099518554780] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 78085 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:54:36,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630150_1099518554780] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 78106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:54:36,788 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630150_1099518554780] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 78259 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 903475, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:54:36,969 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630150_1099518554780] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 78415 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047786, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:54:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=461022, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 12:54:37,357 INFO [MobFileCache #0] mob.MobFileCache: MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 2015-10-16 12:54:37,654 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630151_1099518554781] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 79307 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 718923, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:38,751 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630151_1099518554781] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 80678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762993, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 12:54:38,994 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630151_1099518554781] hdfs.DFSClient: Slow ReadProcessor read fields took 145ms (threshold=20ms); ack: seqno: 80814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989698, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:54:39,874 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630152_1099518554782] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 82030 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 901331, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:40,810 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630152_1099518554782] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 82642 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1040451, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:41,115 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630152_1099518554782] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 82780 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898884, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:41,293 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630152_1099518554782] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 82848 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1131821, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:41,479 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630152_1099518554782] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 82967 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1103025, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:41,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630152_1099518554782] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 83010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 990006, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:41,796 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630152_1099518554782] hdfs.DFSClient: Slow ReadProcessor read fields took 210ms (threshold=20ms); ack: seqno: 83017 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1521943, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:41,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630152_1099518554782] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 83021 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868379, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:41,892 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630152_1099518554782] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 83030 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928361, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:42,043 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630152_1099518554782] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 83158 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 851446, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:42,227 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630152_1099518554782] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 83313 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868693, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:54:42,585 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630154_1099518554784] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 83740 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 8208425, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:42,882 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630154_1099518554784] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 84169 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:43,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630154_1099518554784] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 84734 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792542, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:54:43,878 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630155_1099518554785] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 85559 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 764311, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:44,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630155_1099518554785] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 86097 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 715733, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:44,404 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630155_1099518554785] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 86107 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994841, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:44,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630155_1099518554785] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 86377 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 958628, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:44,814 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630155_1099518554785] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 86546 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930126, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:44,983 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630155_1099518554785] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 86701 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1033233, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:45,112 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630155_1099518554785] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 86794 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 951327, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:45,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630155_1099518554785] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 86938 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1364537, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:45,401 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630155_1099518554785] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 86971 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089240, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:45,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630155_1099518554785] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 87123 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054137, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:45,764 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630155_1099518554785] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 87275 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:45,933 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630155_1099518554785] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 87402 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 916024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:46,005 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630155_1099518554785] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 87468 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 686924, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:46,074 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630155_1099518554785] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 87477 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101426, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:46,185 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630155_1099518554785] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 87497 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1073222, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:54:46,406 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630156_1099518554786] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 87647 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1274987, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:46,579 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630156_1099518554786] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 87806 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1251339, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:46,752 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630156_1099518554786] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 87964 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:46,929 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630156_1099518554786] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 88119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 807454, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:47,085 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630156_1099518554786] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 88266 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942339, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:47,374 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630156_1099518554786] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 88589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 795236, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:47,553 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630156_1099518554786] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 88820 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 663389, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:48,471 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630157_1099518554787] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 90197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 973901, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:54:48,905 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630157_1099518554787] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 90847 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 828365, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:54:49,415 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630157_1099518554787] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 91557 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808206, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:54:49,953 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630158_1099518554788] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 92339 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1034484, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:50,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630158_1099518554788] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 92912 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 708652, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:50,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630158_1099518554788] hdfs.DFSClient: Slow ReadProcessor read fields took 188ms (threshold=20ms); ack: seqno: 92919 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1142825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:50,717 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630158_1099518554788] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 93079 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 846875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:50,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630158_1099518554788] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 93239 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:51,060 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630158_1099518554788] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 93390 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1032744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:51,218 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630158_1099518554788] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 93501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908691, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:51,402 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630158_1099518554788] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 93661 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:51,545 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630158_1099518554788] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 93735 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974902, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:54:51,702 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630159_1099518554789] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 93882 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188969, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:51,986 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630159_1099518554789] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 94281 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801582, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:52,437 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630159_1099518554789] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 94874 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964506, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:52,717 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630159_1099518554789] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 95207 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863894, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:52,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630159_1099518554789] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 95337 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2777835, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:53,067 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630159_1099518554789] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 95493 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936375, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:53,206 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630159_1099518554789] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 95638 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 778325, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:53,254 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630159_1099518554789] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 95642 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1140302, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:53,433 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630159_1099518554789] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 95800 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1004016, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:53,620 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 95966 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 998382, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:53,798 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 96118 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1001409, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:53,977 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 96273 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946671, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:54,157 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 96426 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1238052, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:54,335 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 96578 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1106559, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:54,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 145ms (threshold=20ms); ack: seqno: 96711 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963884, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:54,654 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 96727 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111145, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:54,833 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 96890 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1109817, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:54,997 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 97004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072675, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:55,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 97007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994427, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:55,094 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 97016 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1106060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:55,273 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 97173 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1024248, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:55,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 97328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1147615, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:55,646 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 97467 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1199667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:55,833 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 97609 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1263651, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:56,021 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630160_1099518554790] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 97755 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1100645, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:54:56,497 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630161_1099518554791] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 98375 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 680255, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:54:56,874 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#28 33969200/10109452 (336.01%), rate=50762.14 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=unlimited, activeCompactions=1] 2015-10-16 12:54:56,876 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630161_1099518554791] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 98821 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984216, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:54:57,057 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630161_1099518554791] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 98948 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1083495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:54:57,236 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630161_1099518554791] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 99105 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945723, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:54:57,435 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630161_1099518554791] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 99254 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1041470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:54:57,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630161_1099518554791] hdfs.DFSClient: Slow ReadProcessor read fields took 143ms (threshold=20ms); ack: seqno: 99263 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:54:57,661 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630161_1099518554791] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 99269 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:54:57,861 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630161_1099518554791] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 99422 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992055, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:54:57,945 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630161_1099518554791] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 99445 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967006, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:54:58,109 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630161_1099518554791] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 99604 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864012, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:54:58,241 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630161_1099518554791] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 99694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012968, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:54:58,294 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630161_1099518554791] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 99733 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 639959, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:54:58,353 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630161_1099518554791] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 99743 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 871370, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:54:58,532 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630161_1099518554791] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 99886 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869830, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:54:58,711 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630162_1099518554792] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 100058 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1212616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:58,840 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630162_1099518554792] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 100117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1155792, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:59,013 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630162_1099518554792] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 100275 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022221, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:59,184 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630162_1099518554792] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 100423 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1027668, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:59,354 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630162_1099518554792] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 100579 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 978912, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:59,524 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630162_1099518554792] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 100738 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 939123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:59,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630162_1099518554792] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 100860 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047259, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:54:59,918 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630162_1099518554792] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 101088 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 734662, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:00,865 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630163_1099518554793] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 102448 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 696962, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:55:01,636 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630163_1099518554793] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 103516 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790729, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:55:01,859 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630163_1099518554793] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 103796 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5108833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:55:02,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630164_1099518554794] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 104357 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006837, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:55:02,900 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630164_1099518554794] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 105155 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 676905, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:55:03,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 106514 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 754504, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:04,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 106738 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896462, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:04,062 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 106745 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 933364, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:04,108 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 106757 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927761, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:04,153 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 106764 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:04,508 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 351ms (threshold=20ms); ack: seqno: 106772 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1476040, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:04,619 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 106780 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1067756, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:04,947 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 324ms (threshold=20ms); ack: seqno: 106789 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1272717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:05,083 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 106796 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1529946, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:05,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 232ms (threshold=20ms); ack: seqno: 106804 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1203692, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:05,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 106812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1464019, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:05,726 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 286ms (threshold=20ms); ack: seqno: 106820 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1399960, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:06,130 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 302ms (threshold=20ms); ack: seqno: 106957 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1573180, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:06,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 106964 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1238540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:06,385 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 167ms (threshold=20ms); ack: seqno: 106973 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1123273, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:06,421 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 106983 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1028733, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:06,504 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 106987 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026658, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:06,618 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 106995 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989990, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:06,767 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 107004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1904955, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:07,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 246ms (threshold=20ms); ack: seqno: 107010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1409781, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:07,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 107018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:07,323 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 171ms (threshold=20ms); ack: seqno: 107026 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1282491, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:07,384 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 107033 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1235121, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:07,438 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 107038 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1259360, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:07,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 107043 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1330512, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:07,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 107156 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1030476, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:07,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 107161 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920888, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:07,801 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 107169 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1004415, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:07,852 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 107174 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102891, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:08,092 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 239ms (threshold=20ms); ack: seqno: 107177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1482089, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:08,166 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 107184 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188586, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:08,218 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 107185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2597915, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:08,290 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 107195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088112, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:08,402 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 107201 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047579, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:08,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 359ms (threshold=20ms); ack: seqno: 107209 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 69939583, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:08,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 107217 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1290299, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:09,170 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 366ms (threshold=20ms); ack: seqno: 107225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1360786, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:09,218 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 107232 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1621634, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:09,335 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 107306 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7201095, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:09,477 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 107391 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2653947, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:09,510 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 107397 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910354, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:09,626 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 107405 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017141, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:09,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 107459 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081870, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:09,987 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 247ms (threshold=20ms); ack: seqno: 107464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085672, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:10,026 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 107474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922285, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:10,074 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 107480 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954749, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:10,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 107625 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983647, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:10,558 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 272ms (threshold=20ms); ack: seqno: 107632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1577319, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:10,662 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 107639 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1131713, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:10,937 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 272ms (threshold=20ms); ack: seqno: 107647 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1505931, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:11,054 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 107663 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:11,334 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 277ms (threshold=20ms); ack: seqno: 107670 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1307561, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:11,619 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 184ms (threshold=20ms); ack: seqno: 107786 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963735, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:11,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 107857 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 775727, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:11,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 107865 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1084298, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:11,893 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 107874 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1007995, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:12,188 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 291ms (threshold=20ms); ack: seqno: 107881 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1455115, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:12,298 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 107889 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1374969, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:12,594 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 292ms (threshold=20ms); ack: seqno: 107896 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1349426, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:12,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 107903 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1282104, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:13,000 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 297ms (threshold=20ms); ack: seqno: 107911 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1261187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:13,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 107918 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1293442, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:13,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 366ms (threshold=20ms); ack: seqno: 107927 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1365247, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:13,524 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 107933 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1295171, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:13,558 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 107935 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1195938, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:13,861 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 299ms (threshold=20ms); ack: seqno: 107943 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1387900, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:13,972 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 107947 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1270947, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:14,266 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 290ms (threshold=20ms); ack: seqno: 107954 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1455882, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:14,386 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 107963 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1274640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:14,548 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 108106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870541, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:14,619 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 108152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:14,704 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 108160 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 906589, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:14,811 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 108167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954568, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:15,108 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 293ms (threshold=20ms); ack: seqno: 108175 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1333899, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:15,137 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 108180 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1098859, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:15,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 108183 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1142074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:15,532 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 313ms (threshold=20ms); ack: seqno: 108191 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1356754, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:15,811 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 274ms (threshold=20ms); ack: seqno: 108199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1374544, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:15,843 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 108204 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1231560, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:15,964 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 108211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1389294, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:16,105 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 108274 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 916921, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:16,199 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 108280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1030353, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:16,302 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 108288 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1056946, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:16,422 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 108295 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1202653, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:16,542 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 108303 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1157698, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:16,669 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 108310 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:16,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 108316 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:16,849 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 108323 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1124462, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:17,041 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 151ms (threshold=20ms); ack: seqno: 108347 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1126717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:17,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630165_1099518554795] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 108363 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1121810, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:17,283 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 108374 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1454861, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:17,440 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 108389 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1311556, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:17,579 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 108403 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1261648, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:17,709 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 108425 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1198562, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:17,825 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 108538 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014112, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:17,950 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 108552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006562, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:18,088 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 108575 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1080616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:18,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 108584 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1133053, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:18,336 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 108592 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1122716, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:18,418 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 108662 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1136690, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:18,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 108683 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1172252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:18,697 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 108695 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1208599, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:18,820 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 108703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1264588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:18,954 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 108718 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1198407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:19,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 108735 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1238938, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:19,241 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 158ms (threshold=20ms); ack: seqno: 108749 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1226035, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:19,378 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 108758 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1132266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:19,478 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 108766 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994489, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:19,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 108891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1122871, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:19,793 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 108898 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1258212, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:19,905 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 108906 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1199755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:20,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 108913 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:20,122 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 108922 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1205857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:20,227 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 108929 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1239761, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:20,330 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 108936 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1149869, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:20,772 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 330ms (threshold=20ms); ack: seqno: 109082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1562840, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:20,882 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 109091 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1229525, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:21,020 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 109214 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 669974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:21,074 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 109222 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 962835, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:21,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 109349 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1175386, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:21,341 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 109350 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1264552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:21,563 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 217ms (threshold=20ms); ack: seqno: 109357 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1572685, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:21,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 109364 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1386281, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:21,666 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 109365 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1711820, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:21,933 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 262ms (threshold=20ms); ack: seqno: 109374 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1638669, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:22,037 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 109381 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1625266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:22,339 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 297ms (threshold=20ms); ack: seqno: 109389 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 12735218, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:22,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 109397 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1683325, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:22,716 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 281ms (threshold=20ms); ack: seqno: 109404 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1529220, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:22,817 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 109412 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1467618, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:22,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 109419 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1421998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:22,997 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 109426 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1216249, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:23,307 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 306ms (threshold=20ms); ack: seqno: 109435 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1473640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:23,369 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 109441 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1530935, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:23,420 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 109472 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924937, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:23,521 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 109479 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1097515, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:23,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 109486 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1266527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:23,815 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 188ms (threshold=20ms); ack: seqno: 109492 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1158405, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:23,909 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 109496 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1131318, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:23,944 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 109506 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992285, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:24,213 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 267ms (threshold=20ms); ack: seqno: 109508 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1568950, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:24,335 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 109516 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1519955, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:24,637 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 299ms (threshold=20ms); ack: seqno: 109524 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1422709, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:24,756 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 109531 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1467038, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:25,068 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 308ms (threshold=20ms); ack: seqno: 109538 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1764963, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:25,155 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 109540 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1558119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:25,353 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 196ms (threshold=20ms); ack: seqno: 109546 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1612361, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:25,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 109554 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1149496, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:25,775 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 324ms (threshold=20ms); ack: seqno: 109561 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1454694, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:25,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 109569 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1450923, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:26,014 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 109576 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1251034, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:26,342 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 324ms (threshold=20ms); ack: seqno: 109584 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1510512, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:26,392 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 109590 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1137187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:26,469 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 109591 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1300788, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:26,552 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 109608 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021009, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:26,614 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 109620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983808, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:26,694 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 109627 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1093761, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:26,803 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 109635 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989565, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:27,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 313ms (threshold=20ms); ack: seqno: 109643 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1536461, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:27,231 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 109650 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1273245, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:27,313 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 109667 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917684, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:27,347 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 109675 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 14872524, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:27,450 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 109681 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1146544, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:27,585 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 109688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1059890, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:27,625 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 109690 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 8173678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:27,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 245ms (threshold=20ms); ack: seqno: 109696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1308356, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:27,982 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 109703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1681703, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:28,336 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 351ms (threshold=20ms); ack: seqno: 109711 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1478071, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:28,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 199ms (threshold=20ms); ack: seqno: 109719 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1514753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:28,674 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 109727 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062308, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:28,935 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 257ms (threshold=20ms); ack: seqno: 109734 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1608276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:28,964 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 109747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1166226, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:29,057 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 109755 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1179730, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:29,364 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 304ms (threshold=20ms); ack: seqno: 109760 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1556286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:29,391 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 109766 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1251480, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:29,483 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 109768 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1344270, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:29,551 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 109776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1003315, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:29,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 243ms (threshold=20ms); ack: seqno: 109777 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1732692, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:29,927 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 109784 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1453686, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:29,962 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 109791 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1302400, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:30,205 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 243ms (threshold=20ms); ack: seqno: 109792 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1484717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:30,330 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 109807 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2638150, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:30,416 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 109813 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1138565, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:30,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 185ms (threshold=20ms); ack: seqno: 109814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1103360, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:30,714 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 109821 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1138044, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:31,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 274ms (threshold=20ms); ack: seqno: 109830 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1580264, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:31,130 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 109837 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1748744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:31,416 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 268ms (threshold=20ms); ack: seqno: 109844 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1801891, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:31,480 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 109850 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1291843, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:31,539 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 109856 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1355267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:31,574 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 109865 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930532, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:31,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 298ms (threshold=20ms); ack: seqno: 109871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1689156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:32,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 566ms (threshold=20ms); ack: seqno: 109879 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2860808, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:32,672 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 222ms (threshold=20ms); ack: seqno: 109889 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1342702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:32,718 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 109891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1712842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:33,068 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 334ms (threshold=20ms); ack: seqno: 109900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1599447, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:33,146 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 109908 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1336414, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:33,270 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 109931 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1162865, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:33,370 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 109938 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1184443, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:33,487 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 109946 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1145601, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:33,581 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 109957 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1115804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:33,718 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 109972 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1201758, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:33,854 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 109979 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1224669, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:34,140 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 277ms (threshold=20ms); ack: seqno: 109987 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1607324, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:34,268 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 109994 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1720457, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:34,307 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 109997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1172667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:34,612 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 302ms (threshold=20ms); ack: seqno: 110003 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1685623, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:35,633 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 1016ms (threshold=20ms); ack: seqno: 110011 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1978821, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:36,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 481ms (threshold=20ms); ack: seqno: 110019 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2115276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:36,178 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 110027 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1193690, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:36,424 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 241ms (threshold=20ms); ack: seqno: 110035 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1536574, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:36,759 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 319ms (threshold=20ms); ack: seqno: 110043 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1555643, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:36,896 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 110055 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1331944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:36,968 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 110073 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1018785, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:37,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 110081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1073627, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:37,191 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 110088 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1138364, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:37,231 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 110093 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1041389, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:37,327 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 110095 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006970, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:37,354 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 110102 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:37,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 110110 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:37,434 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 110119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830234, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:37,528 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: Waited 90019ms on a compaction to clean up 'too many store files'; waited long enough... proceeding with flush of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:55:37,528 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536924496 > lower bound=16777216 2015-10-16 12:55:37,528 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 12:55:37,582 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 110135 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1096503, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:37,628 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 110142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 938016, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:37,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 16206ms (threshold=20ms); ack: seqno: 18454 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 953013, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:37,662 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 110150 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869167, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:37,808 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 110162 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081036, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:37,951 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 137ms (threshold=20ms); ack: seqno: 110170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1131522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:38,070 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 110177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1093998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:38,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630167_1099518554797] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 746111, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:38,136 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 110183 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1134951, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:38,190 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 110185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 951060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:38,341 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 143ms (threshold=20ms); ack: seqno: 110192 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1025744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:38,487 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 110206 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1195602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:38,605 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 110213 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1151148, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:38,631 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 110217 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864402, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:38,669 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 110229 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896095, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:38,755 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 110234 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1075494, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:38,791 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 110241 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887064, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 12:55:38,919 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 110249 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1150350, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:39,064 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 110261 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866642, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:39,091 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 110269 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 850302, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:39,220 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 110277 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070823, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:39,256 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 110285 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 822386, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:39,386 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 110304 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945659, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:39,502 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 110311 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1051930, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:39,618 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 110319 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895562, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:39,656 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630168_1099518554798] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 3387 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 731709, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:39,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 110326 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:39,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 110334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859739, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:39,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 110344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 855368, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:39,894 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 110349 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989646, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:40,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 110357 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1003747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:40,037 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 110364 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 814778, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:40,123 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 110368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 976907, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:40,163 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 110375 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 850594, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:40,206 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 110384 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011181, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:40,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 110391 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 906920, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:40,276 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 110400 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769862, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:40,344 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 110423 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909952, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:40,352 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=6851383, memsize=512.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d2921ac14c6b44ad874eb8b2923d3e90 2015-10-16 12:55:40,360 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d2921ac14c6b44ad874eb8b2923d3e90 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d2921ac14c6b44ad874eb8b2923d3e90 2015-10-16 12:55:40,372 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d2921ac14c6b44ad874eb8b2923d3e90, entries=1864380, sequenceid=6851383, filesize=283.2 M 2015-10-16 12:55:40,374 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 2733ms (threshold=20ms); ack: seqno: 18455 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 516777, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:40,374 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536924080, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2846ms, sequenceid=6851383, compaction requested=true 2015-10-16 12:55:40,374 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:55:40,374 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 12 store files, 6 compacting, 6 eligible, 10 blocking 2015-10-16 12:55:40,378 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 6 files of size 1708066553 starting at candidate #0 after considering 10 permutations with 3 in ratio 2015-10-16 12:55:40,378 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:55:40,378 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:55:40,378 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 6 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=1.6 G 2015-10-16 12:55:40,378 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a289bcb11d5f4574b597f51f09338490, keycount=5595750, bloomtype=ROW, size=850.0 M, encoding=NONE, seqNum=6338608 2015-10-16 12:55:40,378 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ae4112bd517046cfbd47fbd7accb0055, keycount=466290, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=6385241 2015-10-16 12:55:40,378 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a87966246b014f83994b5da26116f5ea, keycount=1864390, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=6571685 2015-10-16 12:55:40,378 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c65c8eb5eb134ea4917542070ea7f4ba, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=6618314 2015-10-16 12:55:40,378 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/73de7b1a6d184505bea1cc92489215fb, keycount=466230, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=6664941 2015-10-16 12:55:40,378 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d2921ac14c6b44ad874eb8b2923d3e90, keycount=1864380, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=6851383 2015-10-16 12:55:40,408 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:55:40,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 110426 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 931831, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:40,491 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 110434 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:40,527 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 116 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 938667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:40,562 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 124 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 880726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:40,610 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 110439 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1120390, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:40,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 110446 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1158902, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:40,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 391ms (threshold=20ms); ack: seqno: 133 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1353426, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:41,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630166_1099518554796] hdfs.DFSClient: Slow ReadProcessor read fields took 289ms (threshold=20ms); ack: seqno: 110450 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 288581613, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:41,083 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1297580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:41,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 158ms (threshold=20ms); ack: seqno: 110482 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1284260, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:41,218 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 147 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026433, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:41,344 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 156 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1151119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:41,387 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 160 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:41,476 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 254ms (threshold=20ms); ack: seqno: 110491 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1671257, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:41,596 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 110498 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1362089, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:41,622 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 233ms (threshold=20ms); ack: seqno: 163 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1186317, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:41,865 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 265ms (threshold=20ms); ack: seqno: 110506 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1449663, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:42,004 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 110512 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1573532, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:42,014 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 212ms (threshold=20ms); ack: seqno: 460 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1338375, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:42,059 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 110518 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1396325, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:42,099 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 468 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1273099, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:42,192 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 110530 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1115242, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:42,193 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 483 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915467, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:42,294 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 110537 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1121623, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:42,370 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 169ms (threshold=20ms); ack: seqno: 491 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1015220, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:42,396 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 498 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:42,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 225ms (threshold=20ms); ack: seqno: 505 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1472598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:42,654 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 357ms (threshold=20ms); ack: seqno: 110546 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1454247, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:42,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 110551 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3075119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:42,884 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 257ms (threshold=20ms); ack: seqno: 514 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174499, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:42,937 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 531 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 750384, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:42,974 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 242ms (threshold=20ms); ack: seqno: 110557 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1466889, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:43,038 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 537 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954553, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:43,103 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 110565 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1625781, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:43,206 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 110571 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1414840, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:43,232 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 190ms (threshold=20ms); ack: seqno: 544 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994622, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:43,389 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 182ms (threshold=20ms); ack: seqno: 110572 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1524478, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:43,499 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 110579 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1079613, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:43,786 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 280ms (threshold=20ms); ack: seqno: 110586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1300222, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:43,896 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 110593 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1336449, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:44,081 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630170_1099518554800] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 1846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672774, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:55:44,224 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 320ms (threshold=20ms); ack: seqno: 110602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1387244, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:44,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 110609 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1436565, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:44,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 2084 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1994737, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:44,564 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 2548 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 719591, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:44,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 273ms (threshold=20ms); ack: seqno: 110617 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1283190, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:44,607 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 2554 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992363, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:44,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 2562 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1065198, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:44,666 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 110624 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1195876, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:44,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 138ms (threshold=20ms); ack: seqno: 2570 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932946, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:44,812 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 2577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865502, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:44,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 2586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975941, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:44,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 188ms (threshold=20ms); ack: seqno: 110634 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089198, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:44,990 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 110646 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1200348, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:45,099 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 110655 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1295135, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:45,137 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 110661 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 999182, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:45,241 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 391ms (threshold=20ms); ack: seqno: 2594 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1140170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:45,293 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 2605 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:45,386 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 2609 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930583, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:45,402 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 265ms (threshold=20ms); ack: seqno: 110663 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1526964, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:45,437 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 2618 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1134338, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:45,547 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 134ms (threshold=20ms); ack: seqno: 110671 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1363772, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:45,655 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 215ms (threshold=20ms); ack: seqno: 2624 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1314268, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:45,710 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 2641 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:45,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 232ms (threshold=20ms); ack: seqno: 110679 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1361454, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:45,898 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 110713 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1040614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:46,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 110735 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1256763, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:46,037 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 323ms (threshold=20ms); ack: seqno: 2647 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1228147, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:46,074 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 2655 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1456689, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:46,274 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 206ms (threshold=20ms); ack: seqno: 110785 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1323484, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:46,490 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 182ms (threshold=20ms); ack: seqno: 110814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1134537, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:46,490 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 213ms (threshold=20ms); ack: seqno: 2968 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1465575, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:46,615 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 110823 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020148, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:46,617 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 2977 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1132109, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:46,741 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 110842 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017297, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:46,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 3000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 997099, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:46,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 110853 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1082282, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:47,138 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 111157 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985402, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:47,338 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 111303 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085437, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:47,356 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 196ms (threshold=20ms); ack: seqno: 3657 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986731, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:47,436 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 3665 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:47,520 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 3725 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 670418, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:47,520 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 111433 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736780, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:47,613 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 111442 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928185, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:47,719 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 111452 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1027329, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:47,737 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 3997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924205, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:47,779 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 111463 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 939069, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:47,840 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630172_1099518554802] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 4110 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 851360, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:47,940 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 111546 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011350, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:48,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 111576 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1126298, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:48,183 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 111585 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1181268, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:48,309 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 111606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1271959, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:48,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 111656 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:48,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 111732 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965583, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:48,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 111785 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1098025, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:48,854 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 111819 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1220338, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:48,985 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 111828 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1134520, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:49,202 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 111987 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1116468, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:49,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 111998 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1150524, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:49,423 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 112008 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1032584, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:49,538 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 112021 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1049361, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:49,549 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630173_1099518554803] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 4254 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830233, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:55:49,567 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 112030 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 822074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:49,676 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630173_1099518554803] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 4386 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762106, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:55:49,676 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 112119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 713566, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:49,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630173_1099518554803] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 4432 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 28551759, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:55:49,807 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 112136 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:49,822 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630173_1099518554803] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 4527 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1051020, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:55:49,897 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 112147 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1036097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:49,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 9589ms (threshold=20ms); ack: seqno: 18456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 579718, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:49,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 112163 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 973354, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:50,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 18488 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 677869, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:50,126 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630173_1099518554803] hdfs.DFSClient: Slow ReadProcessor read fields took 143ms (threshold=20ms); ack: seqno: 4768 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943826, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:55:50,127 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 112193 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:50,159 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630173_1099518554803] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 4774 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908863, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:55:50,161 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 18490 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1007212, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:50,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 112230 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988729, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:50,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 137ms (threshold=20ms); ack: seqno: 18495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 581698, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:50,350 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630173_1099518554803] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 4893 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 957483, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:55:50,352 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 112262 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1042162, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:50,435 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 18497 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 569966, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:50,448 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 112293 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988713, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:50,572 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 18504 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 639751, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:50,582 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 112314 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 981417, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:50,591 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630173_1099518554803] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 5184 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 802009, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:55:50,631 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 112323 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037793, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:50,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630173_1099518554803] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 5297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 771600, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:55:50,710 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 18510 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 460021, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:50,809 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630173_1099518554803] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 5407 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 714458, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:55:50,833 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 112436 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950095, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:50,853 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 143ms (threshold=20ms); ack: seqno: 18511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 562680, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:50,947 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 18524 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 685149, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:50,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 112455 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013975, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:50,981 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630173_1099518554803] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 5561 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:55:50,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 18534 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 626540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:51,014 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630171_1099518554801] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 112464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994591, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:55:51,045 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 18547 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 522222, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:51,139 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 18562 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 631067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:51,187 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 18566 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 681556, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:51,233 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 112602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006963, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:51,240 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 18598 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 641372, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:51,330 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 18604 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 475065, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:51,350 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 112608 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1030995, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:51,472 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 112628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941071, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:51,593 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 112649 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1090454, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:51,722 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 112682 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 999263, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:51,760 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 112686 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812819, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:51,760 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 19912 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 447891, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:51,760 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:55:51,760 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:55:51,760 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:55:51,760 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:55:51,823 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 19950 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969763, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:51,876 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 112702 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1189148, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:51,925 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 19960 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 607192, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:51,979 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 112712 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940438, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:52,005 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 19997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 663352, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:52,100 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 112721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1248510, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:52,130 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 20011 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 636586, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:52,419 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 314ms (threshold=20ms); ack: seqno: 112732 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1308406, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:52,595 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 112777 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1206498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:52,746 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 142ms (threshold=20ms); ack: seqno: 112789 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1171737, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:52,897 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 145ms (threshold=20ms); ack: seqno: 112800 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1090301, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:52,931 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 112809 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1025368, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:53,001 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 112829 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014741, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:53,181 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 147ms (threshold=20ms); ack: seqno: 112852 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1131847, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:53,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 112861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1122044, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:53,277 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 112881 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070228, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:53,420 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 138ms (threshold=20ms); ack: seqno: 112892 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1141029, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:53,573 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 147ms (threshold=20ms); ack: seqno: 112903 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1109098, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:53,669 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 112941 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 747286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:53,806 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 112946 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1002562, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:53,840 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 112956 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:53,959 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630173_1099518554803] hdfs.DFSClient: Slow ReadProcessor read fields took 2540ms (threshold=20ms); ack: seqno: 6231 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1090647, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:55:53,962 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 112966 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 977535, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:54,040 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 113000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 894029, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:54,184 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 138ms (threshold=20ms); ack: seqno: 113009 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048759, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:54,191 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630175_1099518554805] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 6573 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902231, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:54,236 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 113021 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:54,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630175_1099518554805] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 6761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2502213, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:54,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 113127 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 881567, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:54,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 23006 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 881570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:54,474 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 113138 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013938, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:54,491 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630175_1099518554805] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 6825 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927226, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:54,528 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 113148 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 867907, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:54,857 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 113391 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1115436, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:54,859 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630175_1099518554805] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 7202 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1033998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:54,990 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 23597 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 471035, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:54,990 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:55:54,990 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 12:55:54,990 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:55:54,990 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 12:55:54,990 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 12:55:54,997 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630175_1099518554805] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 7259 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 24005921, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:55,035 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 113549 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888735, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:55,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 23782 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:55,267 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 113693 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044277, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:55,289 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630175_1099518554805] hdfs.DFSClient: Slow ReadProcessor read fields took 221ms (threshold=20ms); ack: seqno: 7405 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1003183, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:55,334 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:55:55,334 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 113709 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 826810, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:55,336 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630175_1099518554805] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 7413 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964535, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:55,342 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 23827 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7323411, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:55,439 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630175_1099518554805] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 7421 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1190009, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:55,439 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 113720 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127789, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:55,493 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 113742 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:55,622 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 113762 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043959, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:55,636 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630175_1099518554805] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 7640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 944165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:55,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 113771 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:55,810 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630175_1099518554805] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 7825 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 36291584, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:55,880 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 113814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 890385, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:56,010 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 113831 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1076633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:56,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630175_1099518554805] hdfs.DFSClient: Slow ReadProcessor read fields took 151ms (threshold=20ms); ack: seqno: 7960 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 803274, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:56,285 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 113966 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072881, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:56,338 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 24651 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728201, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:56,478 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 186ms (threshold=20ms); ack: seqno: 113977 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1094490, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:56,614 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 113987 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1114632, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:56,906 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#28 5162825/10723290 (48.15%), rate=15466.90 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=unlimited, activeCompactions=2] 2015-10-16 12:55:56,909 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 273ms (threshold=20ms); ack: seqno: 114002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1377723, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:56,977 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 114020 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992291, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:57,113 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 114031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1032266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:57,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630176_1099518554806] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 9576 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 702321, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:55:57,214 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 25291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1285554, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:57,257 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 114041 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1001271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:57,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 25330 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 507145, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:57,393 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 114063 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1145677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:57,566 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 114120 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:57,701 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 114154 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 953741, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:57,736 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 114164 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868621, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:57,853 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 114218 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 867133, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:57,954 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630177_1099518554807] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 10637 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 678663, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:57,954 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 25988 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 401712, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:57,954 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:55:57,954 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:55:57,954 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:55:57,980 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 114228 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1052771, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:58,151 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 114248 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1063542, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:58,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 114286 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060549, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:58,440 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 114307 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050618, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:58,514 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 114344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 740491, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:58,514 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630177_1099518554807] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 11381 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 739615, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:58,514 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 28129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 537529, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:58,514 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:55:58,514 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:55:58,514 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:55:58,514 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:55:58,514 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:55:58,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 114347 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1100953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:58,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 114357 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1027557, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:58,834 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 114368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1067852, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:58,993 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 114396 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1039673, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:59,061 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630177_1099518554807] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 12104 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 694991, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:55:59,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 114405 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1167907, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:59,082 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 142ms (threshold=20ms); ack: seqno: 30462 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 788155, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:59,190 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 114420 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066172, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:59,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 114442 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898455, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:59,332 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 114478 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 962520, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:59,394 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 114499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 997335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:59,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 12585 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1018485, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:55:59,498 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 12596 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1100806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:55:59,509 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 114537 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012603, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:59,607 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 33374 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 491444, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:55:59,608 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:55:59,608 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:55:59,608 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:55:59,608 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:55:59,608 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:55:59,609 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 12604 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1112448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:55:59,629 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 114549 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085268, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:59,655 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 12615 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930441, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:55:59,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 146ms (threshold=20ms); ack: seqno: 114558 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070258, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:59,803 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 12632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:55:59,893 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630174_1099518554804] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 114606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:55:59,947 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 114628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1133359, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:00,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 114644 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1226913, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:00,123 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 316ms (threshold=20ms); ack: seqno: 12639 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1315220, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:00,123 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 114653 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1171655, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:00,222 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 12647 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1055206, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:00,312 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 37567 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 523169, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:00,312 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 12703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 800367, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:00,312 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:00,312 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:00,312 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:00,312 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:00,312 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:00,313 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 114793 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:00,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 12712 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969787, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:00,432 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 114803 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:00,502 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 12720 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1113784, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:00,521 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 114888 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 740092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:00,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 114896 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801055, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:00,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 115022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975542, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:00,812 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 207ms (threshold=20ms); ack: seqno: 12843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1392465, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:00,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 12939 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 775459, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:00,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630144_1099518554774] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 41166 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 501659, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:00,932 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:00,932 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:00,932 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:00,932 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:00,933 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 191ms (threshold=20ms); ack: seqno: 115028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994226, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:01,050 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 12955 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1077996, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:01,253 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 198ms (threshold=20ms); ack: seqno: 115164 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1158440, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:01,258 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:56:01,292 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:56:01,326 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025257980 with entries=78046, filesize=122.35 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025361258 2015-10-16 12:56:01,328 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025241331 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025241331 2015-10-16 12:56:01,329 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630180_1099518554810] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008832, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:01,334 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025250119 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025250119 2015-10-16 12:56:01,368 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 115173 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 796026, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:01,384 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 13299 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 906229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:01,518 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:01,518 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630180_1099518554810] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 813 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 690678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:01,518 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 13424 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 768775, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:01,518 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:01,518 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:01,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 147ms (threshold=20ms); ack: seqno: 115182 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 889185, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:01,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 115203 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 933336, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:01,803 DEBUG [PriorityRpcServer.handler=16,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:56:01,803 WARN [MemStoreFlusher.0] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:56:01,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:56:01,829 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 115335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 756481, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:01,948 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 115350 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751682, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:01,964 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 13868 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 962381, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:02,054 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 13942 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 753338, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:02,054 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630180_1099518554810] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 3501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 669425, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:02,054 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:02,054 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:02,054 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:02,054 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:02,054 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:02,056 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 115362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 999020, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:02,088 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 115373 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044525, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:02,118 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 115384 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862064, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:02,337 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 218ms (threshold=20ms); ack: seqno: 13998 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1213726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:02,338 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 198ms (threshold=20ms); ack: seqno: 115404 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 851376, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:02,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 115413 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 957906, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:02,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 206ms (threshold=20ms); ack: seqno: 14130 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1274030, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:02,661 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 115535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 931893, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:02,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630180_1099518554810] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 7295 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 540106, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:02,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 115606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 675018, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:02,766 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:02,766 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:02,766 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:02,766 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:02,766 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:02,768 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 14138 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1447584, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:03,001 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 229ms (threshold=20ms); ack: seqno: 14145 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1108882, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:03,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 226ms (threshold=20ms); ack: seqno: 115618 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 883510, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:03,103 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 14154 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1211445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:03,153 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 115744 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869489, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:03,281 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 115760 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 923576, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:03,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 14335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1125980, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:03,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630180_1099518554810] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 10942 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 527886, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:03,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 14445 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:03,414 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:03,414 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:03,414 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:03,414 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:03,415 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 115771 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 901817, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:03,634 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 205ms (threshold=20ms); ack: seqno: 14458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1387695, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:03,637 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 115908 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 931673, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:03,645 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630180_1099518554810] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 11589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085727, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:03,734 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630178_1099518554808] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 14467 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1148000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:03,834 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 116034 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965783, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:04,055 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 215ms (threshold=20ms); ack: seqno: 14610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1103452, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:04,056 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 116168 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 916455, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:04,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 14651 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 651086, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:04,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630180_1099518554810] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 14070 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 413302, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:04,128 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:04,128 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:04,128 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:04,128 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:04,128 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:04,130 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 116178 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1064152, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:04,243 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 14657 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:04,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 116187 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912645, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:04,344 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 14664 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:04,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 213ms (threshold=20ms); ack: seqno: 14700 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1381385, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:04,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 116461 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950820, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:04,701 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 14708 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1126140, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:04,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630180_1099518554810] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 17593 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 566228, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:04,787 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:04,787 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:04,787 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:04,787 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:04,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 149ms (threshold=20ms); ack: seqno: 116602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1436557, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:04,836 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 14723 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1873080, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:04,934 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 116613 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 848088, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:04,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 116618 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1003944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:05,070 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 14971 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 21002381, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:05,162 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 185ms (threshold=20ms); ack: seqno: 116630 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 976249, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:05,163 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 14975 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081282, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:05,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 116640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1140261, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:05,395 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:05,395 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:05,395 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 15220 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847780, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:05,395 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:05,395 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630180_1099518554810] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 20716 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 481642, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:05,395 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:05,395 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:05,423 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630179_1099518554809] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 116693 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2812760, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:05,554 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 116717 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1275432, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:05,569 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 15362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 802797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:05,638 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 116726 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1266381, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:05,703 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 15481 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 899607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:05,927 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 209ms (threshold=20ms); ack: seqno: 116812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:05,927 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 15616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:05,961 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 116823 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 781369, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:06,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630180_1099518554810] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 23801 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 686490, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:06,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 15670 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 824674, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:06,018 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:06,018 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:06,018 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:06,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 116848 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769347, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:06,018 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:06,018 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:06,160 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 116866 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 816821, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:06,172 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 141ms (threshold=20ms); ack: seqno: 15687 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 880724, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:06,251 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 116876 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:06,331 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 15834 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 51087641, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:06,496 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 223ms (threshold=20ms); ack: seqno: 116898 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1278814, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:06,497 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 15960 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 916639, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:06,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 117027 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794194, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:06,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630180_1099518554810] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 27020 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 499130, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:06,647 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:06,647 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:06,647 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:06,647 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:06,648 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630181_1099518554811] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 16073 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917544, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:06,772 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 117042 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1040716, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:06,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 117048 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070087, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:06,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 117060 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 894627, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:07,020 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 117063 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:07,052 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 117068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:07,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 117099 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 846289, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:07,196 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630180_1099518554810] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 29710 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 580700, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:07,196 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630183_1099518554813] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 16689 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2870393, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:56:07,197 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:56:07,197 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:56:07,197 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:56:07,197 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:56:07,197 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:56:07,198 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 117117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 959447, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:07,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 117155 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1117974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:07,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 117161 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1087128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:07,490 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 117168 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 961330, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:07,573 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 117174 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:07,618 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 117178 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 931080, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:07,684 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 117184 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 970534, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:07,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630183_1099518554813] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 17357 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728674, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:56:07,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630180_1099518554810] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 32269 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489915, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:07,730 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:07,730 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:07,730 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:07,731 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:07,762 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 117188 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986161, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:07,842 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 117192 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1046824, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:07,955 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 117197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 802743, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:08,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 117207 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:08,175 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 117211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 980674, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:08,287 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630183_1099518554813] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 18036 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 667454, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:56:08,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 117216 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1124946, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:08,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 117224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909529, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:08,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630180_1099518554810] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 34934 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 784276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:08,440 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 117236 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812241, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:08,484 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 117242 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942003, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:08,548 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630183_1099518554813] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 18216 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971790, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:56:08,622 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 117280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940518, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:08,640 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630183_1099518554813] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 18313 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:56:08,672 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630183_1099518554813] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 18317 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830694, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:56:08,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 117318 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866314, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:08,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 117334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 918913, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:08,874 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630183_1099518554813] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 18501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:56:08,888 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 117353 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943999, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:08,956 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630183_1099518554813] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 18551 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 768681, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:56:08,956 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630180_1099518554810] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 37837 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 463128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:08,956 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:08,956 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:08,956 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:08,956 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:08,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 117359 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 960291, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:09,050 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 117362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1032388, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:09,103 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 117366 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:09,199 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:56:09,245 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:56:09,267 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025361258 with entries=78126, filesize=122.47 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025369199 2015-10-16 12:56:09,270 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111208, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:09,378 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 272ms (threshold=20ms); ack: seqno: 117372 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1170659, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:09,425 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 117393 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1187106, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:09,494 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 19222 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811716, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:09,494 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 1022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 502880, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:09,494 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:09,494 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:09,494 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:09,494 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:09,496 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 117398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050658, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:09,619 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 117404 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 977529, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:09,710 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 117409 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:09,749 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 117425 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:09,834 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 117430 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1039989, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:09,955 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 117450 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011685, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:10,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 19824 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 732951, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:10,028 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:56:10,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 3842 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 516066, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:10,028 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:56:10,028 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:56:10,028 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:56:10,029 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 12:56:10,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 117462 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966851, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:10,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 117470 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982969, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:10,161 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 117475 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 843139, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:10,197 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 117484 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 933571, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:10,251 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 117494 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1075419, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:10,260 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 19973 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833085, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:10,387 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 19987 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937347, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:10,406 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 117513 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 959422, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:10,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 117523 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006533, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:10,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 138ms (threshold=20ms); ack: seqno: 19996 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 791931, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:10,549 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 117533 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1103981, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:10,666 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 20003 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 938812, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:10,690 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 134ms (threshold=20ms); ack: seqno: 117544 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062631, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:10,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 8174 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 393222, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:10,787 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:10,787 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:10,787 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:10,787 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:10,788 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 117555 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 772935, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:10,821 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 20027 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857961, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:10,829 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 117559 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 777483, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:10,964 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 117582 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964233, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:10,973 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 146ms (threshold=20ms); ack: seqno: 20035 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839159, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:11,086 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 117591 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 906123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:11,099 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 20043 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 816218, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:11,126 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 20051 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 727627, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:11,199 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 117610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 944614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:11,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 20054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888416, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:11,326 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 117620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937629, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:11,346 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 20062 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 871230, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:11,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 117630 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895332, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:11,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 117640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1197009, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:11,498 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 147ms (threshold=20ms); ack: seqno: 20070 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1034519, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:11,557 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 117670 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869603, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:11,616 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 20078 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989333, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:11,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 13031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 541053, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:11,672 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:11,672 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:11,672 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:11,672 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:11,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 117692 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905027, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:11,764 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 117708 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982432, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:11,851 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 230ms (threshold=20ms); ack: seqno: 20086 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088693, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:11,867 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 117712 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 739143, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:12,097 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 220ms (threshold=20ms); ack: seqno: 20101 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1180880, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:12,123 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 235ms (threshold=20ms); ack: seqno: 117732 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1073506, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:12,269 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 117750 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 837056, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:12,366 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 265ms (threshold=20ms); ack: seqno: 20109 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1287357, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:12,414 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 117760 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888294, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:12,492 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 117775 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006474, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:12,525 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 155ms (threshold=20ms); ack: seqno: 20116 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1354705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:12,590 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 17873 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819225, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:12,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 18041 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 452181, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:12,667 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:56:12,667 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:56:12,667 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:56:12,667 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:56:12,667 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:56:12,668 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 174ms (threshold=20ms); ack: seqno: 117780 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896542, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:12,770 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 117791 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 837864, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:12,824 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 117797 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 953787, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:12,877 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 348ms (threshold=20ms); ack: seqno: 20123 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066394, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:12,883 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 117807 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 809045, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:12,944 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 117812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 939002, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:12,954 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 20132 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1090621, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:13,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 117843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905316, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:13,262 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 117854 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1117445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:13,289 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 330ms (threshold=20ms); ack: seqno: 20140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1146529, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:13,390 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 117863 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913581, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:13,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 20147 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044379, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:13,427 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 117873 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941928, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:13,542 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 23099 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 400717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:13,542 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:56:13,542 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:56:13,542 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:56:13,542 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:56:13,542 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:56:13,563 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 117887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 900389, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:13,636 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 117908 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902674, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:13,734 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 330ms (threshold=20ms); ack: seqno: 20155 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1113734, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:13,771 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 117917 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965105, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:13,815 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 117929 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804848, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:13,820 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 20163 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1138886, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:13,921 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 20170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054812, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:13,970 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 117945 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1055097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:14,273 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 347ms (threshold=20ms); ack: seqno: 20178 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1163452, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:14,347 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 370ms (threshold=20ms); ack: seqno: 117956 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1360777, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:14,408 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 20187 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1028248, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:14,469 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 28227 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 517229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:14,469 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:56:14,469 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:56:14,469 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:56:14,469 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:56:14,469 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:56:14,491 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 117967 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896205, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:14,693 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 281ms (threshold=20ms); ack: seqno: 20194 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1204470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:14,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 297ms (threshold=20ms); ack: seqno: 117977 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1109085, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:14,803 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 20202 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060963, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:14,888 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 117986 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1129433, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:15,015 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 117991 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1087953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:15,140 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 332ms (threshold=20ms); ack: seqno: 20210 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 18896226, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:15,191 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 173ms (threshold=20ms); ack: seqno: 117996 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:15,209 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 20217 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1046803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:15,283 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 118004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836396, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:15,343 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 118011 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1136744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:15,411 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:56:15,411 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:56:15,411 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:56:15,411 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:56:15,411 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:56:15,478 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 264ms (threshold=20ms); ack: seqno: 20225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1353843, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:15,579 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 235ms (threshold=20ms); ack: seqno: 118014 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1354623, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:15,585 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 20234 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207783, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:15,627 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 20237 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1356149, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:15,642 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 118020 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1045129, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:15,666 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 118024 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 774444, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:15,687 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 20242 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948026, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:15,727 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 118028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943783, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:15,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 20249 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863972, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:15,821 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 118046 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 990882, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:15,966 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 221ms (threshold=20ms); ack: seqno: 20258 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1515153, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:16,039 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 215ms (threshold=20ms); ack: seqno: 118051 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1617059, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:16,042 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 20265 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1082085, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:16,102 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 118057 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1206170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:16,140 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 118062 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:16,148 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 20269 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1298229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:16,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 20277 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 952093, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:16,203 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 118068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895873, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:16,333 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 20336 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1093791, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:16,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 198ms (threshold=20ms); ack: seqno: 118074 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1238778, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:16,421 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 20343 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043260, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:16,471 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 118080 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1057638, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:16,563 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 118106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196318, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:16,594 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 118113 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 973357, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:16,711 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 118138 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1148243, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:16,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 134ms (threshold=20ms); ack: seqno: 118147 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1232769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:17,036 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 118270 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111322, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:17,123 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 118279 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1079795, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:17,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 118301 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013878, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:17,375 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 118311 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1061191, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:17,488 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 118322 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1228501, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:17,595 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 118332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946235, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:17,716 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 118342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:17,748 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 118353 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1114247, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:17,968 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 118503 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 960718, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:18,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 118512 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1208170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:18,188 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 118522 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1091332, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:18,307 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 118533 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1209023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:18,428 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 118547 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048257, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:18,542 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 118566 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1075177, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:18,644 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 118577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1041734, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:18,683 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 118586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 952740, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:18,751 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 118589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:18,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 118593 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037416, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:18,908 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 118603 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1121862, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:18,989 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 118609 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1297187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:19,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 118617 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1039751, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:19,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 118624 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 894826, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:19,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 118641 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920519, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:19,230 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 118650 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 8132362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:19,492 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 251ms (threshold=20ms); ack: seqno: 118658 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1529859, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:19,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 118665 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1052994, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:19,828 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 241ms (threshold=20ms); ack: seqno: 118673 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1354282, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:19,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 118680 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1288636, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:20,207 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 265ms (threshold=20ms); ack: seqno: 118688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1472981, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:20,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 118696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1416722, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:20,494 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 190ms (threshold=20ms); ack: seqno: 118703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1520762, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:20,618 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 118711 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 6695134, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:20,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 262ms (threshold=20ms); ack: seqno: 118718 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1737297, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:20,915 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 118726 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1216991, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:21,042 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 118733 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037153, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:21,261 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 214ms (threshold=20ms); ack: seqno: 118741 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1530149, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:21,396 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 118748 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1582344, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:21,564 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 164ms (threshold=20ms); ack: seqno: 118757 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1104128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:21,666 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 118766 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1205465, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:21,760 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 118774 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1299543, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:21,793 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630182_1099518554812] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 118781 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:56:21,855 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 118801 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 961714, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:22,109 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 250ms (threshold=20ms); ack: seqno: 118808 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1498092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:22,202 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 118815 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1152674, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:22,445 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 239ms (threshold=20ms); ack: seqno: 118823 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1261767, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:22,563 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 118830 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1135063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:22,681 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 6260ms (threshold=20ms); ack: seqno: 20344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1710221, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:22,824 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 256ms (threshold=20ms); ack: seqno: 118838 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1296938, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:22,830 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 20431 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1018908, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:22,917 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 118846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1214234, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:23,013 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 20628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 722516, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:23,015 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 118854 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1128750, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:23,250 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 230ms (threshold=20ms); ack: seqno: 20636 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1497058, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:23,250 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 212ms (threshold=20ms); ack: seqno: 118872 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1061358, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:23,367 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 118913 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838046, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:23,515 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 160ms (threshold=20ms); ack: seqno: 20760 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1200317, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:23,516 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 145ms (threshold=20ms); ack: seqno: 118920 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896596, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:23,599 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630184_1099518554814] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 20766 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964449, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:56:23,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 192ms (threshold=20ms); ack: seqno: 118998 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1145649, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:23,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 20891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1625131, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:23,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 119005 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1092940, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:24,052 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 189ms (threshold=20ms); ack: seqno: 21010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1462093, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:24,053 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 119130 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985277, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:24,146 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 21068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1074981, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:24,153 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 119135 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 883962, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:24,264 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 21075 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1270106, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:24,266 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 119142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 978386, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:24,314 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 119149 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023796, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:24,382 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 119158 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 889034, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:24,437 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 119163 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915543, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:24,438 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 21206 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:24,585 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 119186 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089746, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:24,586 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 137ms (threshold=20ms); ack: seqno: 21220 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1246754, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:24,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 21300 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 939533, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:24,672 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 119193 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 851753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:24,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 21307 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1376660, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:24,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 119201 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050971, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:24,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 119210 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 854755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:24,877 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 119215 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 933837, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:24,905 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 21409 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 939123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:24,948 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 119226 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934176, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:25,059 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 21429 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1318336, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:25,061 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 119234 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 939719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:25,112 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 119242 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 800428, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:25,167 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 21501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1090810, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:25,211 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 119257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 886729, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:25,279 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 21506 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1316965, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:25,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 119265 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 860514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:25,311 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 21514 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1094294, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:25,377 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 21557 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 770360, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:25,379 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 119272 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808006, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:25,406 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 119276 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 745362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:25,477 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 119281 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909847, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:25,588 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 119297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 980454, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:25,705 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 119306 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904728, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:25,751 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 119324 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983994, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:25,841 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 119326 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085487, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:25,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 119331 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 978297, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:25,976 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 119336 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 997235, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:26,115 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 725ms (threshold=20ms); ack: seqno: 21572 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1789160, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:26,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 119354 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021752, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:26,261 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 142ms (threshold=20ms); ack: seqno: 21580 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1443883, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:26,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 119373 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1003985, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:26,321 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 21588 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1291029, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:26,419 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 119439 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 999955, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:26,627 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 119586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 959584, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:26,628 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 204ms (threshold=20ms); ack: seqno: 21726 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2450272, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:26,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 21735 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1258478, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:26,945 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 216ms (threshold=20ms); ack: seqno: 119736 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1200478, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:26,947 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 21888 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1151552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:27,050 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 119743 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2813707, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:27,071 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 22032 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927224, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:27,295 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 211ms (threshold=20ms); ack: seqno: 119774 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1202218, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:27,296 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 22193 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174459, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:27,397 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 119782 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1543329, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:27,477 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 119834 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 670728, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:27,479 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 22352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 903418, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:27,574 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 22361 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 877293, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:27,580 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 119841 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 996817, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:27,753 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 119898 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 883830, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:27,888 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 22598 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1121937, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:27,889 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 119930 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102694, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:28,162 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 245ms (threshold=20ms); ack: seqno: 22612 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1641719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:28,163 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 214ms (threshold=20ms); ack: seqno: 119985 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:28,251 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 119996 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1018621, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:28,301 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 22733 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787356, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:28,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 160ms (threshold=20ms); ack: seqno: 120050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1079724, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:28,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630187_1099518554817] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 22854 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919836, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:56:28,546 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 120059 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 852708, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:28,834 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 291ms (threshold=20ms); ack: seqno: 22960 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1205116, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:28,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 212ms (threshold=20ms); ack: seqno: 120166 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967617, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:28,921 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 22969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971410, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:29,077 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 166ms (threshold=20ms); ack: seqno: 120279 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1075153, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:29,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 23090 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864441, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:29,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 120287 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941658, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:29,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 120327 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692159, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:29,217 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 23210 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1045286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:29,428 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 207ms (threshold=20ms); ack: seqno: 120335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2776580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:29,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 208ms (threshold=20ms); ack: seqno: 23217 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3526283, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:29,502 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 120342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972429, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:29,661 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 157ms (threshold=20ms); ack: seqno: 23327 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:29,662 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 120438 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1106763, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:29,746 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 23335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866376, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:29,806 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 120522 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 881550, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:30,033 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 220ms (threshold=20ms); ack: seqno: 23419 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1370181, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:30,035 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 207ms (threshold=20ms); ack: seqno: 120528 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1259221, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:30,185 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 120536 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1263079, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:30,197 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 23498 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 976652, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:30,240 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 23505 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:30,302 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 120544 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967864, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:30,377 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 23589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 894861, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:30,473 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 23644 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 980461, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:30,511 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 23650 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:30,511 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 120652 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048249, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:30,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 120658 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 961397, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:30,569 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 23667 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885204, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:30,648 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 120664 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085186, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:30,649 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 23673 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1201002, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:30,706 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 120673 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 976200, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:30,715 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 23690 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 824063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:30,901 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 192ms (threshold=20ms); ack: seqno: 120679 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1180431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:30,902 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 183ms (threshold=20ms); ack: seqno: 23695 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089767, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:30,929 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 120687 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873099, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:30,971 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 23714 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1252993, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:31,010 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 120695 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982183, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:31,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 23723 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1157856, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:31,070 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 120701 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1123533, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:31,076 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 23737 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 860185, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:31,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 120707 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971942, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:31,115 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 23745 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 903323, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:31,172 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 120717 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941768, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:31,177 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 23761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1078642, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:31,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 187ms (threshold=20ms); ack: seqno: 120723 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1138537, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:31,364 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 182ms (threshold=20ms); ack: seqno: 23769 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1355158, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:31,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 23784 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037548, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:31,490 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630186_1099518554816] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 120764 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038662, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:31,491 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 23788 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1244895, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:31,570 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 23793 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864710, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:31,684 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 23819 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874471, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:31,883 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 208ms (threshold=20ms); ack: seqno: 120992 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1685954, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:31,892 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 206ms (threshold=20ms); ack: seqno: 23824 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1155242, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:31,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 121000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1080480, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:31,981 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 121042 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 785633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:31,987 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 23842 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:32,037 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 121049 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969469, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:32,038 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 23848 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984457, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:32,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 121056 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769206, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:32,154 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 23982 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839599, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:32,217 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 121068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 918506, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:32,238 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 24004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102561, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:32,325 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 24028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 944803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:32,412 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 24052 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006397, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:32,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 225ms (threshold=20ms); ack: seqno: 121076 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1257484, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:32,510 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 24074 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:32,527 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 121084 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1164896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:32,612 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 121091 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 785336, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:32,675 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 121097 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902491, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:32,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 24228 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1314012, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:32,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 121140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836132, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:32,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 121147 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946742, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:32,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 24251 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011695, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:32,861 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 121161 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926319, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:32,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 24273 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967311, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:32,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 24295 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 740577, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:32,973 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 121216 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 918832, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:33,068 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 24320 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922243, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:33,102 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 121228 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 49413570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:33,112 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 24327 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 892886, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:33,176 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 24344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971455, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:33,222 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 121308 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887722, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:33,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 24369 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 961766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:33,311 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 24374 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1316558, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:33,366 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 121416 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 933828, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:33,371 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 24394 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 780514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:33,450 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 24398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924506, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:33,510 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 121535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 952880, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:33,592 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 24440 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7848125, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:33,661 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 121642 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1034840, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:33,679 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 24463 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769102, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:33,706 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 24467 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787204, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:33,768 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 24506 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782366, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:33,769 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 121709 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941720, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:33,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 121717 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 814175, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:33,910 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 121725 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869302, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:33,962 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 121734 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874733, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:34,020 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 121741 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 821415, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:34,130 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 24784 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813197, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:34,221 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 121891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 852944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:34,336 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 24947 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5692888, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:34,392 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630188_1099518554818] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 24992 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 25794495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:34,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 122040 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:34,495 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 122052 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1181098, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:34,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 122059 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 901964, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:34,734 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 122074 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1161979, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:34,761 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 122079 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 809896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:34,825 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 122085 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895204, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:34,841 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 25354 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 33227097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:34,900 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 122141 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 33894339, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:34,900 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 25380 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 25884791, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:34,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 122153 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154121, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:35,001 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 25478 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 33261137, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,021 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 122200 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 33145430, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:35,041 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 25508 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 35356187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,059 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 122220 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 32765814, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:35,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 122237 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 36353639, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:35,159 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 25600 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 31599525, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,211 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 25639 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 27783312, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,265 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 25681 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 29773256, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,316 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 25718 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 40167825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,351 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 25748 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 29984009, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,391 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 25777 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 34781409, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,424 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 25798 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 30808717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,441 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 319ms (threshold=20ms); ack: seqno: 122295 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 321598585, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:35,463 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 25818 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 36513785, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,496 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 25842 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 28956111, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,531 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 122338 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 31647557, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:35,546 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 25878 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 29402450, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,564 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 122362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26977155, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:35,596 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 122378 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 27359138, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:35,611 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 25929 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 27055634, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,626 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 122396 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26573008, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:35,646 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 25954 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 24402848, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,681 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 25981 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 25925688, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,721 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 122456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26496416, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:35,752 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 26036 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 35983509, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,808 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 26077 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 24190200, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,816 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 122512 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 75579142, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:35,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 122527 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26979539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:35,868 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 26126 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 25383634, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:35,877 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 122537 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 772339, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:35,977 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 122605 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26190755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:36,004 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 122620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 24146004, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:36,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 26242 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 22028938, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:36,037 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 122637 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 29796346, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:36,056 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 26275 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 31185814, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:36,089 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 122646 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 778290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:36,151 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 26288 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:36,180 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 122739 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 29614925, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:36,192 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 26294 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 858619, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:36,230 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 122775 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26088898, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:36,275 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 26367 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 29123866, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:36,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 122798 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 759891, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:36,317 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 26393 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 37735733, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:36,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 26413 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 40306286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:36,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 26434 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 37641424, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:36,400 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 122870 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 28099502, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:36,450 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 26465 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 25488193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:36,454 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#32 average throughput is 28.27 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 12:56:36,460 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630189_1099518554819] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 122901 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 32112723, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:36,511 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630190_1099518554820] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 26506 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 27187017, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:36,533 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hdfs.DFSClient: Slow waitForAckedSeqno took 76ms (threshold=20ms) 2015-10-16 12:56:36,556 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/97dc7feeac8749ef8d85f3ae08fe46a4 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/97dc7feeac8749ef8d85f3ae08fe46a4 2015-10-16 12:56:36,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 21125ms (threshold=20ms); ack: seqno: 33700 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 605246, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:36,572 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:56:36,578 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:56:36,590 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a289bcb11d5f4574b597f51f09338490, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a289bcb11d5f4574b597f51f09338490 2015-10-16 12:56:36,598 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ae4112bd517046cfbd47fbd7accb0055, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ae4112bd517046cfbd47fbd7accb0055 2015-10-16 12:56:36,606 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a87966246b014f83994b5da26116f5ea, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a87966246b014f83994b5da26116f5ea 2015-10-16 12:56:36,615 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c65c8eb5eb134ea4917542070ea7f4ba, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c65c8eb5eb134ea4917542070ea7f4ba 2015-10-16 12:56:36,623 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/73de7b1a6d184505bea1cc92489215fb, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/73de7b1a6d184505bea1cc92489215fb 2015-10-16 12:56:36,631 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d2921ac14c6b44ad874eb8b2923d3e90, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d2921ac14c6b44ad874eb8b2923d3e90 2015-10-16 12:56:36,632 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 6 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 97dc7feeac8749ef8d85f3ae08fe46a4(size=1.6 G), total size for store is 10.2 G. This selection was in queue for 0sec, and took 56sec to execute. 2015-10-16 12:56:36,632 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=6, fileSize=1.6 G, priority=-2, time=36903249232015885; duration=56sec 2015-10-16 12:56:36,632 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:56:36,632 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 6 compacting, 1 eligible, 10 blocking 2015-10-16 12:56:36,633 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:56:36,633 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:56:36,633 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:56:36,716 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 123027 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 40003597, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:36,751 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 123053 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 30290158, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:36,791 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 123087 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 32949002, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:36,817 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 123109 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 22795829, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:36,857 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 123130 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 37220408, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:36,902 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 123164 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 38304449, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:36,976 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 123224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 40223409, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:37,004 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 123250 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 24676491, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:37,015 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536920576 > lower bound=16777216 2015-10-16 12:56:37,016 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 12:56:37,033 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 123270 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26045624, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:37,058 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 123291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 22942536, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:37,115 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 123336 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 27660177, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:37,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 549ms (threshold=20ms); ack: seqno: 33701 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 669140, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:37,155 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 123361 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 35747921, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:37,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 123397 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 33344770, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:37,206 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 0 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 34769997, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:37,227 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 123422 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 27497675, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:37,265 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 123447 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 33810718, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:37,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 123476 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 30058177, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:37,373 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 123515 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 757238, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:37,450 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 123585 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26190630, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:37,465 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 155 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 28627400, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:37,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 27101159, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:37,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 123621 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 38183878, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:37,711 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 300 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 24548129, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:37,725 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 220ms (threshold=20ms); ack: seqno: 123646 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044121, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:37,860 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 123759 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 23249615, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:38,006 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 123765 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:38,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 25279203, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:38,110 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 560 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26991246, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:38,154 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 594 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26047516, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:38,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 706 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 30862962, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:38,335 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 733 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 23679152, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:38,380 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 769 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 36326702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:38,410 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 788 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26358643, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:38,505 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 859 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 30328412, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:38,560 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 897 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 36050994, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:38,634 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 924 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 67661118, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.5714285714285714, tune compaction throughput to 15.71 MB/sec 2015-10-16 12:56:39,041 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 833ms (threshold=20ms); ack: seqno: 123933 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1349722, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:39,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 124034 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 30133248, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:39,200 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 124060 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 31082194, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:39,238 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 124087 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 33623107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:39,322 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 687ms (threshold=20ms); ack: seqno: 925 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 754782981, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:39,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 978 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 28055776, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:39,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 1015 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26360170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:39,437 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 186ms (threshold=20ms); ack: seqno: 124108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1236244, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:39,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1049 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 22989284, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:39,516 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#28 sleep 264 ms because current throughput is 26.88 MB/sec, max allowed is 15.71 MB/sec, already slept 75 time(s) and total slept time is 44768 ms till now. 2015-10-16 12:56:39,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 1103 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 24504770, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:39,547 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 124201 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 39234913, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:39,585 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 1138 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 27221471, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:39,635 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 1174 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 30209091, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:39,690 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1226 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 22292777, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:39,763 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 1282 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 25594583, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:39,782 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 225ms (threshold=20ms); ack: seqno: 124221 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1480918, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:39,855 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 1353 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26501562, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:39,885 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 1375 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 25974911, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:39,920 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 1396 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 31106419, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:39,953 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 124258 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044379, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:40,084 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 124264 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1075501, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:40,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630192_1099518554822] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1750 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5443282, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:40,449 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 349ms (threshold=20ms); ack: seqno: 124273 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1430036, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:40,788 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 332ms (threshold=20ms); ack: seqno: 124279 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1353556, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:40,869 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 124285 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1446069, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:40,870 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630193_1099518554823] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 2868 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1389888, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:41,047 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 124393 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:41,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 124468 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983316, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:41,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630193_1099518554823] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 4148 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 705090, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:56:41,612 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 362ms (threshold=20ms); ack: seqno: 124557 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1237709, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:41,656 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 124562 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044642, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:41,806 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 124654 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922002, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:41,840 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=7037824, memsize=512.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b17d9013d33a4b3ca0e5d11e7777c1c0 2015-10-16 12:56:41,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b17d9013d33a4b3ca0e5d11e7777c1c0 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b17d9013d33a4b3ca0e5d11e7777c1c0 2015-10-16 12:56:41,860 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b17d9013d33a4b3ca0e5d11e7777c1c0, entries=1864360, sequenceid=7037824, filesize=283.2 M 2015-10-16 12:56:41,862 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 4741ms (threshold=20ms); ack: seqno: 33702 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 622697, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:41,862 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536920160, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 4847ms, sequenceid=7037824, compaction requested=false 2015-10-16 12:56:42,247 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 388ms (threshold=20ms); ack: seqno: 124724 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1469363, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:42,308 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 124728 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1397842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:42,506 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 124855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1119350, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:42,884 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 355ms (threshold=20ms); ack: seqno: 124891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1591818, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:42,975 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630191_1099518554821] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 124966 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:56:43,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 467ms (threshold=20ms); ack: seqno: 125061 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1456787, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:43,640 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 1778ms (threshold=20ms); ack: seqno: 33703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 735118, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:44,006 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 35654 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 620123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:44,006 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:56:44,006 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:56:44,006 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:56:44,006 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 12:56:44,011 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:56:44,156 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 521ms (threshold=20ms); ack: seqno: 125228 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1211414, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:44,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 125327 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913401, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:44,411 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 125335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 923307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:44,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 39128 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792647, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:44,791 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 336ms (threshold=20ms); ack: seqno: 125395 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1328902, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:44,796 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630185_1099518554815] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 39247 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 615705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:44,828 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:56:44,873 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025369199 with entries=78023, filesize=122.31 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025404828 2015-10-16 12:56:44,875 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025257980 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025257980 2015-10-16 12:56:44,876 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630196_1099518554826] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942604, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:44,881 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025361258 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025361258 2015-10-16 12:56:44,908 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 125409 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921217, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:45,046 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630196_1099518554826] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 703761, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:45,046 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 12:56:45,046 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 12:56:45,046 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 12:56:45,046 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 12:56:45,046 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 12:56:45,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 125430 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1077465, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:45,277 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 125556 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993044, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:45,427 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 143ms (threshold=20ms); ack: seqno: 125561 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1139213, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:45,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 125567 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1033140, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:45,581 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 125574 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:45,725 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 125583 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1148082, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:45,808 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 125591 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885548, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:45,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630196_1099518554826] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 5067 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 753580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:45,929 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 125599 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1859702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:45,972 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630196_1099518554826] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 5519 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 607634, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:45,972 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 125608 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804130, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:45,972 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:45,972 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:45,972 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:45,972 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:45,972 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:46,087 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 125619 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983091, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:46,301 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 125730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963661, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:46,435 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 125746 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922642, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:46,535 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 125754 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943146, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:46,818 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630196_1099518554826] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 9991 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 508422, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:46,818 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:56:46,818 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:56:46,818 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:46,818 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:56:46,818 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:56:46,976 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 337ms (threshold=20ms); ack: seqno: 125877 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1143154, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:47,089 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 125885 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060919, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:47,306 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 126007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1040672, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:47,411 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 126015 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1003808, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:47,523 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 126022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813815, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:47,670 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630196_1099518554826] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 14922 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 544018, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:47,670 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:47,671 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:47,671 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:47,671 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:47,671 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:47,672 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 126031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985639, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:47,797 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 126045 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948308, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:47,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 126053 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 878580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:48,029 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 126060 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885403, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:48,094 DEBUG [PriorityRpcServer.handler=10,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:56:48,094 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134258976 > lower bound=16777216 2015-10-16 12:56:48,095 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:56:48,121 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630196_1099518554826] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 17588 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 594554, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:48,139 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 126068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1181630, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:48,349 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 126174 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1106742, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:48,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630197_1099518554827] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 351 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 737211, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:56:48,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630196_1099518554826] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 19114 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 455370, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:48,443 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:48,443 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:48,443 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:48,443 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:48,443 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:48,468 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 126182 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 903118, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:48,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 164ms (threshold=20ms); ack: seqno: 126231 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1185358, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:48,720 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630196_1099518554826] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 19953 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 651168, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:48,757 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 126236 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985183, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:48,861 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 126244 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022634, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:48,964 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=7084456, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/9f6f9dac76b642b38b216e8c1ba7fa4d 2015-10-16 12:56:48,969 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/9f6f9dac76b642b38b216e8c1ba7fa4d as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9f6f9dac76b642b38b216e8c1ba7fa4d 2015-10-16 12:56:48,975 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9f6f9dac76b642b38b216e8c1ba7fa4d, entries=466280, sequenceid=7084456, filesize=70.8 M 2015-10-16 12:56:48,976 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134284480, currentsize=20.90 MB/21918880 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 882ms, sequenceid=7084456, compaction requested=true 2015-10-16 12:56:48,976 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:56:48,977 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 6 compacting, 3 eligible, 10 blocking 2015-10-16 12:56:48,978 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:56:48,978 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:56:48,978 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:56:49,073 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 126365 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988768, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:49,170 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630196_1099518554826] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 22279 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 620205, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:49,189 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 126374 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 935648, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:49,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 126398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:49,418 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 126405 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1083982, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:49,516 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 126410 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 953960, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:49,619 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 126418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947792, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:49,939 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 197ms (threshold=20ms); ack: seqno: 126565 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915586, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:50,004 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630196_1099518554826] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 26960 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 514879, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:50,004 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 126602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 750896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:50,005 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:50,005 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:50,005 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:50,005 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:50,005 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:50,575 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 470ms (threshold=20ms); ack: seqno: 126732 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1130452, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:50,788 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630196_1099518554826] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 31294 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 484422, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:50,788 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:50,788 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:50,788 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:50,788 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:51,211 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 496ms (threshold=20ms); ack: seqno: 126899 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1152063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:51,658 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:51,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630196_1099518554826] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 36210 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 478246, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:51,659 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:56:51,659 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:51,659 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:51,659 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:51,847 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630195_1099518554825] hdfs.DFSClient: Slow ReadProcessor read fields took 476ms (threshold=20ms); ack: seqno: 127068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1252587, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:52,124 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:56:52,164 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025404828 with entries=77927, filesize=122.16 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025412124 2015-10-16 12:56:52,166 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025369199 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025369199 2015-10-16 12:56:52,346 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630199_1099518554829] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 570769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:52,484 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 506ms (threshold=20ms); ack: seqno: 127238 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1128769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:52,597 DEBUG [PriorityRpcServer.handler=2,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:56:52,597 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134267664 > lower bound=16777216 2015-10-16 12:56:52,597 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:56:52,597 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 127242 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1236043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:52,657 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 127258 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729371, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:52,672 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630199_1099518554829] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 1827 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 657090, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:53,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 343ms (threshold=20ms); ack: seqno: 127405 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1167309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:53,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630200_1099518554830] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 805 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864336, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:56:53,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 127522 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:53,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630199_1099518554829] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 4663 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 802580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:53,245 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:53,245 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:53,245 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:53,245 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:53,246 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:53,489 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=7131087, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/3e2521c284ec49c686e1e1d27d4accf1 2015-10-16 12:56:53,495 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/3e2521c284ec49c686e1e1d27d4accf1 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3e2521c284ec49c686e1e1d27d4accf1 2015-10-16 12:56:53,500 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3e2521c284ec49c686e1e1d27d4accf1, entries=466270, sequenceid=7131087, filesize=70.8 M 2015-10-16 12:56:53,502 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134281360, currentsize=23.63 MB/24782000 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 905ms, sequenceid=7131087, compaction requested=true 2015-10-16 12:56:53,502 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:56:53,502 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 6 compacting, 4 eligible, 10 blocking 2015-10-16 12:56:53,504 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio 2015-10-16 12:56:53,504 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:56:53,504 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:56:53,755 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 453ms (threshold=20ms); ack: seqno: 127571 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1224724, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:53,888 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 127697 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 757851, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:53,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630199_1099518554829] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 8626 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 562878, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:53,996 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:56:53,996 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:56:53,996 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:56:53,996 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:56:53,996 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:56:54,391 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 474ms (threshold=20ms); ack: seqno: 127738 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1080422, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:54,893 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630199_1099518554829] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 13536 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 525078, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:54,894 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:54,894 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:54,894 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:54,894 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:54,894 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:55,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 487ms (threshold=20ms); ack: seqno: 127905 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1249601, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:55,664 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 483ms (threshold=20ms); ack: seqno: 128072 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1228341, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:55,749 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 128129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769038, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:55,749 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630199_1099518554829] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 18175 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 624409, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:55,749 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:55,749 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:55,750 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:55,750 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:55,750 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:56,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 466ms (threshold=20ms); ack: seqno: 128239 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1332424, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:56,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630199_1099518554829] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 22604 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 540436, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:56,572 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:56,572 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:56,572 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:56,572 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:56,572 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:56,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630199_1099518554829] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 23667 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 621287, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:56,935 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#28 17981644/15776574 (113.98%), rate=14677.79 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=15.71 MB/sec, activeCompactions=1] 2015-10-16 12:56:56,937 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 506ms (threshold=20ms); ack: seqno: 128408 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1606557, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:56,971 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 128414 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1387668, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:57,074 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630199_1099518554829] hdfs.DFSClient: Slow ReadProcessor read fields took 199ms (threshold=20ms); ack: seqno: 23668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 569095, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:57,074 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 200ms (threshold=20ms) 2015-10-16 12:56:57,074 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 200ms (threshold=20ms) 2015-10-16 12:56:57,074 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 200ms (threshold=20ms) 2015-10-16 12:56:57,074 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 200ms (threshold=20ms) 2015-10-16 12:56:57,074 INFO [sync.0] wal.FSHLog: Slow sync cost: 200 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:57,074 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 199ms (threshold=20ms) 2015-10-16 12:56:57,074 INFO [sync.2] wal.FSHLog: Slow sync cost: 199 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:57,075 INFO [sync.4] wal.FSHLog: Slow sync cost: 199 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:57,074 INFO [sync.3] wal.FSHLog: Slow sync cost: 200 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:57,074 INFO [sync.1] wal.FSHLog: Slow sync cost: 200 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:57,331 DEBUG [PriorityRpcServer.handler=9,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:56:57,331 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134280560 > lower bound=16777216 2015-10-16 12:56:57,331 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:56:57,356 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630199_1099518554829] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 25083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 462995, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:57,572 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 486ms (threshold=20ms); ack: seqno: 128575 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196950, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:57,638 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 128611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1045784, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:57,638 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630199_1099518554829] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 26445 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 573751, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:57,638 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630201_1099518554831] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 386 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 846586, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:56:57,638 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:57,638 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:57,638 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:56:57,638 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:57,638 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:56:58,147 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=7177720, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/43a3e2e93252421e9ffdf7c4cda8d770 2015-10-16 12:56:58,152 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/43a3e2e93252421e9ffdf7c4cda8d770 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/43a3e2e93252421e9ffdf7c4cda8d770 2015-10-16 12:56:58,158 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/43a3e2e93252421e9ffdf7c4cda8d770, entries=466290, sequenceid=7177720, filesize=70.8 M 2015-10-16 12:56:58,160 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134287920, currentsize=23.53 MB/24672320 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 829ms, sequenceid=7177720, compaction requested=true 2015-10-16 12:56:58,160 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:56:58,160 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 6 compacting, 5 eligible, 10 blocking 2015-10-16 12:56:58,162 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222832274 starting at candidate #2 after considering 6 permutations with 1 in ratio 2015-10-16 12:56:58,162 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:56:58,162 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:56:58,162 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 12:56:58,163 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9f6f9dac76b642b38b216e8c1ba7fa4d, keycount=466280, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=7084456 2015-10-16 12:56:58,163 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3e2521c284ec49c686e1e1d27d4accf1, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=7131087 2015-10-16 12:56:58,163 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/43a3e2e93252421e9ffdf7c4cda8d770, keycount=466290, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=7177720 2015-10-16 12:56:58,177 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:56:58,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 471ms (threshold=20ms); ack: seqno: 128742 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1615029, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:58,259 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630199_1099518554829] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 29610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 497253, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:58,259 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 50 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1003080, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:58,259 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 128759 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 939196, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:58,259 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:58,259 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:58,259 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:58,259 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:58,259 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:59,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630199_1099518554829] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 34050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 476187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:59,063 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:56:59,063 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:56:59,063 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:56:59,064 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:56:59,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 1112ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1202557, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:56:59,480 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 1105ms (threshold=20ms); ack: seqno: 128909 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1251259, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:59,544 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 128919 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1359159, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:59,751 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630199_1099518554829] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 37292 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 688888, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:59,752 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 90ms (threshold=20ms) 2015-10-16 12:56:59,752 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 90ms (threshold=20ms) 2015-10-16 12:56:59,752 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 91ms (threshold=20ms) 2015-10-16 12:56:59,752 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 90ms (threshold=20ms) 2015-10-16 12:56:59,752 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 90ms (threshold=20ms) 2015-10-16 12:56:59,753 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 129044 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1193430, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:59,818 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 129054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088629, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:56:59,968 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630199_1099518554829] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 38268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 374043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:56:59,968 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:56:59,968 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:56:59,968 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:56:59,968 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:56:59,968 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:57:00,036 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:57:00,327 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 279ms (threshold=20ms) 2015-10-16 12:57:00,330 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Slow sync cost: 282 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:57:00,339 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025412124 with entries=81081, filesize=127.10 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025420037 2015-10-16 12:57:00,342 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025404828 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025404828 2015-10-16 12:57:00,726 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 1155ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1087528, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:00,753 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 920ms (threshold=20ms); ack: seqno: 129076 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1311266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:00,868 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630203_1099518554833] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 2554 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 8417114, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:00,868 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 472 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 8310058, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:00,868 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:00,868 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630198_1099518554828] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 129147 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5044267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:00,868 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:00,868 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:00,868 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:00,868 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:01,790 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630203_1099518554833] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 7654 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 524508, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:01,790 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:01,790 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:01,790 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:01,790 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:01,790 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:01,845 DEBUG [PriorityRpcServer.handler=8,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:57:01,845 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:57:01,845 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:57:02,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 1035ms (threshold=20ms); ack: seqno: 129246 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1278098, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:02,094 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 1206ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 96867286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:02,657 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630203_1099518554833] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 11487 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 746858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:02,754 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630203_1099518554833] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 11825 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 518194, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:02,754 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:02,754 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:02,754 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:02,754 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:02,754 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:03,270 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 1095ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1122961, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:03,296 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 1118ms (threshold=20ms); ack: seqno: 129413 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1416679, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:03,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 129514 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 871998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:03,480 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 129558 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863960, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:03,540 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630203_1099518554833] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 16023 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 558201, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:03,540 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:03,540 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:03,540 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:03,540 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:03,540 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:04,581 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630203_1099518554833] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 21578 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895452, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:04,581 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 1191ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:04,581 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:04,581 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:04,581 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:04,581 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:04,581 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:04,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 1088ms (threshold=20ms); ack: seqno: 129580 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1249656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:05,375 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630203_1099518554833] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 25775 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 558290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:05,375 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:57:05,375 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:57:05,375 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:57:05,375 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:57:05,375 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:57:05,815 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 1099ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989574, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:05,855 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 1123ms (threshold=20ms); ack: seqno: 129746 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1193032, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:05,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 129754 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 958828, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:06,188 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630203_1099518554833] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 30020 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 567076, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:06,188 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 55ms (threshold=20ms) 2015-10-16 12:57:06,188 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 55ms (threshold=20ms) 2015-10-16 12:57:06,188 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 55ms (threshold=20ms) 2015-10-16 12:57:06,188 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 55ms (threshold=20ms) 2015-10-16 12:57:06,188 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 55ms (threshold=20ms) 2015-10-16 12:57:07,087 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 1142ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044511, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:07,127 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 1127ms (threshold=20ms); ack: seqno: 129915 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1452540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:07,169 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630203_1099518554833] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 35386 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 591082, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:07,169 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 129916 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1767207, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:07,169 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 1224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 807985, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:07,169 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:07,169 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:07,169 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:07,169 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:07,169 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:07,243 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 129951 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 41212589, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:07,243 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630203_1099518554833] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 35520 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 40941260, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:07,243 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:07,243 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:07,243 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:07,243 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:07,243 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:07,243 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 1271 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 41371980, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:07,937 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630203_1099518554833] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 38785 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 750271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:08,007 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:57:08,055 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025420037 with entries=77980, filesize=122.24 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025428008 2015-10-16 12:57:08,058 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630205_1099518554835] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1134582, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:08,171 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:57:08,171 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:57:08,172 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630205_1099518554835] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 433 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 747639, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:08,172 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:57:08,359 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 1101ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1393983, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:08,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 1094ms (threshold=20ms); ack: seqno: 130082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1118958, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:08,424 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 130083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3200952, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:08,589 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 229ms (threshold=20ms); ack: seqno: 1335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 229851077, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:08,937 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630205_1099518554835] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 4661 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 376661, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:08,937 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:08,937 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:08,937 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:08,937 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:08,937 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:09,631 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 983ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1036642, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:09,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 1109ms (threshold=20ms); ack: seqno: 130249 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1110786, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:09,777 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 1657 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782575, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:09,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 130293 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672590, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:09,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630205_1099518554835] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 9437 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 576091, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:09,778 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:09,778 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:09,778 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:09,778 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:09,778 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:10,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630205_1099518554835] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 14534 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 419003, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:10,604 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:10,604 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:10,604 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:10,604 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:10,604 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:10,903 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 1115ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1316154, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:10,943 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 1068ms (threshold=20ms); ack: seqno: 130416 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1110941, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:11,334 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630205_1099518554835] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 18750 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 482677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:11,334 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:57:11,334 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:57:11,334 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:57:11,334 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:57:11,334 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:57:12,183 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 1152ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 8194833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:12,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 1093ms (threshold=20ms); ack: seqno: 130585 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1346585, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:12,268 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 1900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 684531, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:12,268 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630205_1099518554835] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 24083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 432822, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:12,268 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:57:12,268 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:57:12,268 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:57:12,268 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:57:12,268 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:57:12,270 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 130592 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1272930, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:12,680 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630205_1099518554835] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 25691 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 892622, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:13,058 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:57:13,058 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:57:13,058 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 12:57:13,058 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 12:57:13,236 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:57:13,236 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630205_1099518554835] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 28621 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 370298, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:13,236 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:57:13,236 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:57:13,236 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:57:13,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630202_1099518554832] hdfs.DFSClient: Slow ReadProcessor read fields took 1109ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188909, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:57:13,487 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 1088ms (threshold=20ms); ack: seqno: 130752 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1063339, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:13,760 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:57:13,760 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:57:13,760 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 12:57:14,075 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630205_1099518554835] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 32800 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 411989, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:14,075 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:57:14,075 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:57:14,075 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:57:14,075 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:57:14,075 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:57:14,719 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630206_1099518554836] hdfs.DFSClient: Slow ReadProcessor read fields took 1155ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1331214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:14,759 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 1101ms (threshold=20ms); ack: seqno: 130919 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1116717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:14,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 131019 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:14,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630205_1099518554835] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 37261 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 610449, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:14,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630206_1099518554836] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 2335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825231, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:14,876 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:14,876 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:14,876 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:14,876 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:15,250 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:57:15,299 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:57:15,306 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630205_1099518554835] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 39444 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 574318, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:15,321 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025428008 with entries=77625, filesize=121.69 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025435250 2015-10-16 12:57:15,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630206_1099518554836] hdfs.DFSClient: Slow ReadProcessor read fields took 1114ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1640429, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:16,032 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 1105ms (threshold=20ms); ack: seqno: 131086 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1326451, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:16,068 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 131092 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1502811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:16,156 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 131139 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 32814361, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:16,201 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 131153 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 41501731, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:16,240 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 131166 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 36682949, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:16,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 131180 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 37320244, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:16,321 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 131197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 36471665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:16,361 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 131210 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 37991685, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:17,263 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630206_1099518554836] hdfs.DFSClient: Slow ReadProcessor read fields took 1162ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1809534, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:17,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630204_1099518554834] hdfs.DFSClient: Slow ReadProcessor read fields took 876ms (threshold=20ms); ack: seqno: 131252 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1015338, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:17,460 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630208_1099518554838] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 131291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 28072670, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:57:18,536 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630206_1099518554836] hdfs.DFSClient: Slow ReadProcessor read fields took 1151ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1725266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:18,605 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630206_1099518554836] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 2710 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 973178, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:18,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630208_1099518554838] hdfs.DFSClient: Slow ReadProcessor read fields took 848ms (threshold=20ms); ack: seqno: 131423 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1255485, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:57:19,808 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630206_1099518554836] hdfs.DFSClient: Slow ReadProcessor read fields took 1077ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1537547, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:19,880 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630208_1099518554838] hdfs.DFSClient: Slow ReadProcessor read fields took 914ms (threshold=20ms); ack: seqno: 131590 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1158290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:57:21,082 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630206_1099518554836] hdfs.DFSClient: Slow ReadProcessor read fields took 1159ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3764831, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:21,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630208_1099518554838] hdfs.DFSClient: Slow ReadProcessor read fields took 1085ms (threshold=20ms); ack: seqno: 131757 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1223671, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:57:22,351 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630206_1099518554836] hdfs.DFSClient: Slow ReadProcessor read fields took 1164ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1625524, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:22,423 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630208_1099518554838] hdfs.DFSClient: Slow ReadProcessor read fields took 1014ms (threshold=20ms); ack: seqno: 131923 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1214451, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:57:22,463 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630208_1099518554838] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 131929 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 16149831, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:57:23,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630206_1099518554836] hdfs.DFSClient: Slow ReadProcessor read fields took 1156ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1462896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:23,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630206_1099518554836] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 3375 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 600169, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:23,694 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630208_1099518554838] hdfs.DFSClient: Slow ReadProcessor read fields took 993ms (threshold=20ms); ack: seqno: 132090 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072463, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:57:23,740 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#33 average throughput is 8.08 MB/sec, slept 20 time(s) and total slept time is 22430 ms. 1 active compactions remaining, total limit is 15.71 MB/sec 2015-10-16 12:57:23,760 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/56c38b9cfa9941139a45c7108bf7a330 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/56c38b9cfa9941139a45c7108bf7a330 2015-10-16 12:57:23,776 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 8478ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 594262, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:23,776 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:57:23,781 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:57:23,787 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9f6f9dac76b642b38b216e8c1ba7fa4d, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9f6f9dac76b642b38b216e8c1ba7fa4d 2015-10-16 12:57:23,795 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3e2521c284ec49c686e1e1d27d4accf1, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3e2521c284ec49c686e1e1d27d4accf1 2015-10-16 12:57:23,803 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/43a3e2e93252421e9ffdf7c4cda8d770, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/43a3e2e93252421e9ffdf7c4cda8d770 2015-10-16 12:57:23,804 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 56c38b9cfa9941139a45c7108bf7a330(size=212.5 M), total size for store is 10.6 G. This selection was in queue for 0sec, and took 25sec to execute. 2015-10-16 12:57:23,804 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=-1, time=36903327016343937; duration=25sec 2015-10-16 12:57:23,804 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:57:23,804 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 6 compacting, 3 eligible, 10 blocking 2015-10-16 12:57:23,805 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:57:23,805 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:57:23,805 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:57:24,332 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630208_1099518554838] hdfs.DFSClient: Slow ReadProcessor read fields took 456ms (threshold=20ms); ack: seqno: 132257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2116640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:57:24,383 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536924336 > lower bound=16777216 2015-10-16 12:57:24,383 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 12:57:24,488 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 711ms (threshold=20ms); ack: seqno: 2 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 668381, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:24,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630208_1099518554838] hdfs.DFSClient: Slow ReadProcessor read fields took 440ms (threshold=20ms); ack: seqno: 132423 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1094190, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:57:25,473 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630209_1099518554839] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 1684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 675513, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:25,603 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630208_1099518554838] hdfs.DFSClient: Slow ReadProcessor read fields took 442ms (threshold=20ms); ack: seqno: 132592 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1264931, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:57:26,239 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630208_1099518554838] hdfs.DFSClient: Slow ReadProcessor read fields took 492ms (threshold=20ms); ack: seqno: 132761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1292143, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:57:26,764 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630210_1099518554840] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 3908 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 637353, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:26,876 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630208_1099518554838] hdfs.DFSClient: Slow ReadProcessor read fields took 525ms (threshold=20ms); ack: seqno: 132928 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1310883, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:57:27,195 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=7364163, memsize=512.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a7521413dc0444ce9a26ef76febb7aee 2015-10-16 12:57:27,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a7521413dc0444ce9a26ef76febb7aee as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a7521413dc0444ce9a26ef76febb7aee 2015-10-16 12:57:27,216 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a7521413dc0444ce9a26ef76febb7aee, entries=1864380, sequenceid=7364163, filesize=283.2 M 2015-10-16 12:57:27,217 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 2730ms (threshold=20ms); ack: seqno: 3 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 697871, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:27,218 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536923920, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2835ms, sequenceid=7364163, compaction requested=true 2015-10-16 12:57:27,218 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:57:27,218 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 6 compacting, 4 eligible, 10 blocking 2015-10-16 12:57:27,220 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 816813131 starting at candidate #1 after considering 3 permutations with 1 in ratio 2015-10-16 12:57:27,220 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:57:27,220 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:57:27,221 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=779.0 M 2015-10-16 12:57:27,221 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b17d9013d33a4b3ca0e5d11e7777c1c0, keycount=1864360, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=7037824 2015-10-16 12:57:27,221 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/56c38b9cfa9941139a45c7108bf7a330, keycount=1398840, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=7177720 2015-10-16 12:57:27,221 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a7521413dc0444ce9a26ef76febb7aee, keycount=1864380, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=7364163 2015-10-16 12:57:27,237 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:57:27,522 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630208_1099518554838] hdfs.DFSClient: Slow ReadProcessor read fields took 530ms (threshold=20ms); ack: seqno: 133095 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 11839188, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:57:28,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630212_1099518554842] hdfs.DFSClient: Slow ReadProcessor read fields took 1171ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1693970, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:28,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630208_1099518554838] hdfs.DFSClient: Slow ReadProcessor read fields took 1139ms (threshold=20ms); ack: seqno: 133262 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1430857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:57:29,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630212_1099518554842] hdfs.DFSClient: Slow ReadProcessor read fields took 1168ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1301907, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:30,056 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 1152ms (threshold=20ms); ack: seqno: 133433 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1343206, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:30,090 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 133437 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1029046, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:30,157 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 133488 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 34519041, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:30,182 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 133501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 24367709, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:30,217 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 133517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 33926603, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:30,262 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 133536 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 42828618, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:30,299 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 133555 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 34948743, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:30,337 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 133576 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 35458545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:30,382 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 133599 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 37584662, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:31,064 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630212_1099518554842] hdfs.DFSClient: Slow ReadProcessor read fields took 1157ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1784811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:31,328 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 945ms (threshold=20ms); ack: seqno: 133600 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1461119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:31,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 133643 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 59754060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:31,551 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 133684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 38180486, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:31,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 133705 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 47325431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:31,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 133723 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 38438281, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:31,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 133737 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 44784608, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:31,726 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 133757 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 37645145, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:32,336 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630212_1099518554842] hdfs.DFSClient: Slow ReadProcessor read fields took 1172ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1786673, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:32,600 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 870ms (threshold=20ms); ack: seqno: 133767 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1403756, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:32,653 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 133772 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1460047, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:32,771 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 133860 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 28878102, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:32,801 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 133882 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26393098, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:32,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 133900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 28145941, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:33,420 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 6203ms (threshold=20ms); ack: seqno: 4 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921816, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:33,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630212_1099518554842] hdfs.DFSClient: Slow ReadProcessor read fields took 1172ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1148257, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:33,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630212_1099518554842] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972068, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:33,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 1595 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 468954, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:33,742 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:57:33,742 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:57:33,742 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:57:33,742 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:57:33,742 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:57:33,871 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 1011ms (threshold=20ms); ack: seqno: 133932 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1190585, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:34,557 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 6332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 472444, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:34,557 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:34,557 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:34,557 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:34,557 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:34,557 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:34,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630212_1099518554842] hdfs.DFSClient: Slow ReadProcessor read fields took 1115ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1306173, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:35,142 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 1074ms (threshold=20ms); ack: seqno: 134099 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1027792, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:35,175 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 134106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081258, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:35,337 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 134192 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 28955276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:35,337 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 10731 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 507805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:35,337 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:35,337 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:35,337 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:57:35,337 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:35,337 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:57:35,375 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 134227 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 27073143, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:35,411 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 134251 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 31926966, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:35,505 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 10987 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 785705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:36,153 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630212_1099518554842] hdfs.DFSClient: Slow ReadProcessor read fields took 1147ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1401508, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:36,274 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 15610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 471118, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:36,274 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:57:36,274 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:36,274 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630212_1099518554842] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 1273 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 829620, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:36,274 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:57:36,274 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:36,274 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:36,415 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 998ms (threshold=20ms); ack: seqno: 134266 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127018, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:36,453 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 134271 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1126854, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:37,065 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 20230 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 478663, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:37,065 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:37,065 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:37,065 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:57:37,065 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:37,065 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:57:37,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630212_1099518554842] hdfs.DFSClient: Slow ReadProcessor read fields took 1110ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 8104354, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:37,596 DEBUG [PriorityRpcServer.handler=8,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:57:37,597 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134282016 > lower bound=16777216 2015-10-16 12:57:37,597 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:57:37,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 23589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 441268, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:37,687 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 1056ms (threshold=20ms); ack: seqno: 134433 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1133294, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:37,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 134441 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1278309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:37,811 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630214_1099518554844] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 200 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835490, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:57:37,811 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 134471 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819401, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:37,811 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 24450 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 435141, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:37,812 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:37,812 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:37,812 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:37,812 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:37,812 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:38,397 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630214_1099518554844] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 1083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 777589, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:57:38,397 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 27764 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 557526, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:38,397 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:38,397 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:38,397 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:38,397 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:38,397 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:38,444 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=7410798, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/4440a49a5fa342439b2ef0dd7454b02b 2015-10-16 12:57:38,449 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/4440a49a5fa342439b2ef0dd7454b02b as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4440a49a5fa342439b2ef0dd7454b02b 2015-10-16 12:57:38,455 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4440a49a5fa342439b2ef0dd7454b02b, entries=466310, sequenceid=7410798, filesize=70.8 M 2015-10-16 12:57:38,457 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134293120, currentsize=24.57 MB/25763840 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 860ms, sequenceid=7410798, compaction requested=false 2015-10-16 12:57:38,696 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630212_1099518554842] hdfs.DFSClient: Slow ReadProcessor read fields took 1144ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1239902, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 12:57:38,959 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 1049ms (threshold=20ms); ack: seqno: 134600 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070427, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:38,990 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 134608 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865005, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:39,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 31801 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 596594, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:39,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 134763 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 717907, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:39,133 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:39,133 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:39,134 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:39,133 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:39,134 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:39,740 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 168ms (threshold=20ms); ack: seqno: 135337 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1217608, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:39,753 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 34724 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 570616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:39,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 34837 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 514379, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:39,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630213_1099518554843] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 135387 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 897643, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:39,831 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:39,831 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:39,831 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:39,831 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:39,831 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:39,969 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630212_1099518554842] hdfs.DFSClient: Slow ReadProcessor read fields took 1145ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1211161, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:39,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630215_1099518554845] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 135532 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:57:40,279 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630212_1099518554842] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 1989 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 763842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:40,279 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 37195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 554571, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:40,279 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630215_1099518554845] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 135839 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 671534, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:57:40,279 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:40,279 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:40,279 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:40,279 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:40,279 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:40,628 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:57:40,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630215_1099518554845] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 136280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 734382, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:57:40,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630216_1099518554846] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 2451 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1208611, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:57:40,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630207_1099518554837] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 39106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 479774, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:40,667 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:40,667 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:40,667 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:40,667 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:40,667 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:40,695 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025435250 with entries=77781, filesize=121.93 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025460628 2015-10-16 12:57:40,696 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025412124 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025412124 2015-10-16 12:57:40,703 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025420037 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025420037 2015-10-16 12:57:40,711 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025428008 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025428008 2015-10-16 12:57:41,051 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 1762 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 470384, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:41,051 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630216_1099518554846] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 2899 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808411, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:57:41,051 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630215_1099518554845] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 136721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762601, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:57:41,051 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:41,051 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:41,051 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:41,051 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:41,051 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:41,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630216_1099518554846] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 3336 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 703508, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:57:41,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630215_1099518554845] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 137193 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 706187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:57:41,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 3558 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 379649, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:41,446 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:41,446 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:41,446 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:41,446 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:41,844 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630216_1099518554846] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 3772 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782013, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:57:41,844 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 5384 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 555821, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:41,844 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:41,844 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 137677 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 899023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:41,844 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:41,844 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:57:41,844 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:41,844 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:57:42,230 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 138114 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1000621, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:42,230 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630219_1099518554849] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 4223 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751056, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:42,230 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 7233 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 788800, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:42,230 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:57:42,231 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:57:42,231 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:57:42,231 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:57:42,231 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:57:42,319 WARN [MemStoreFlusher.0] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:57:42,319 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:57:42,319 DEBUG [PriorityRpcServer.handler=12,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:57:42,518 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 138373 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1064165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:42,580 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 138388 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 768090, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:42,662 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630219_1099518554849] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 4730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1553446, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:42,662 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 9456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1483188, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:42,662 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:57:42,662 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:57:42,662 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:57:42,662 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:57:42,662 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:57:42,710 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 138403 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853679, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:42,837 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 138410 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 959498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:42,973 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 138418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930695, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:43,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 138426 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 996343, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:43,035 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 138434 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845153, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:43,071 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 138441 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937090, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:43,194 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:43,194 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 12027 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 591381, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:43,194 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630219_1099518554849] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 5398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 854930, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:43,194 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:43,194 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:43,194 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:43,194 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:43,299 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 225ms (threshold=20ms); ack: seqno: 138445 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1257262, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:43,625 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 322ms (threshold=20ms); ack: seqno: 138452 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1260173, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:43,737 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630219_1099518554849] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 6078 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838749, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:57:43,737 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 14677 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 670239, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:43,737 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:43,737 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:43,737 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:43,737 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:43,737 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:43,761 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 138460 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1226601, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:43,972 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 191ms (threshold=20ms); ack: seqno: 138472 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910396, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:44,116 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 138481 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089026, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:44,174 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 138483 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054240, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:44,276 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 17393 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 609909, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:44,276 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630220_1099518554850] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 6749 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787658, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:57:44,276 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:44,276 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:44,276 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:44,276 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:44,277 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:44,536 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 18134 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 669690, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:44,553 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 358ms (threshold=20ms); ack: seqno: 138493 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1385144, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:44,701 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 18759 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 706375, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:44,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 138530 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1036462, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:45,046 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 202ms (threshold=20ms); ack: seqno: 19375 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 671838, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:45,046 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630220_1099518554850] hdfs.DFSClient: Slow ReadProcessor read fields took 202ms (threshold=20ms); ack: seqno: 7580 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 750156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:57:45,047 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 203ms (threshold=20ms) 2015-10-16 12:57:45,047 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 203ms (threshold=20ms) 2015-10-16 12:57:45,047 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 202ms (threshold=20ms) 2015-10-16 12:57:45,047 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 202ms (threshold=20ms) 2015-10-16 12:57:45,047 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 203ms (threshold=20ms) 2015-10-16 12:57:45,047 INFO [sync.1] wal.FSHLog: Slow sync cost: 202 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:45,047 INFO [sync.3] wal.FSHLog: Slow sync cost: 203 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:45,047 INFO [sync.0] wal.FSHLog: Slow sync cost: 203 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:45,047 INFO [sync.4] wal.FSHLog: Slow sync cost: 203 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:45,047 INFO [sync.2] wal.FSHLog: Slow sync cost: 202 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:45,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 306ms (threshold=20ms); ack: seqno: 138537 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1375429, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:45,175 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 138545 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1419659, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:45,239 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 138551 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924706, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:45,283 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 138552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965369, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:45,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 138556 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1087072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:45,472 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 138567 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1140308, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:45,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 138570 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1098196, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:45,569 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:45,569 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:45,569 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 21815 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 542026, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:45,570 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:45,570 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:45,570 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:45,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630220_1099518554850] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 8288 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1040823, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:57:45,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 138577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:45,679 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 138582 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986930, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:45,740 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 138591 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1025046, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:45,867 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 138599 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 999185, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:45,928 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 138606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1046686, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:46,039 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 138611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069147, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:46,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630221_1099518554851] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 8965 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701660, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:57:46,078 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:46,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 138615 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:46,078 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:46,078 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:46,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 24448 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672426, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:46,078 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:46,078 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:46,230 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 138638 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 906872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:46,350 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 25914 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811573, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:46,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 138646 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1164293, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:46,567 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 26863 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 605991, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:46,567 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:46,567 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:46,567 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:46,567 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:46,567 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:46,567 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630221_1099518554851] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 9552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 738353, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:57:46,759 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 270ms (threshold=20ms); ack: seqno: 138791 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1355572, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:46,851 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 138799 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 960300, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:46,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 138806 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921206, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:47,039 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 138814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1046786, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:47,101 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 29534 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 560111, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:47,101 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630221_1099518554851] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 10222 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779776, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:57:47,101 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:47,101 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:47,101 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:47,101 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:47,101 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:47,281 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 238ms (threshold=20ms); ack: seqno: 138822 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1428366, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:47,364 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 138829 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1225840, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:47,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 138837 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985593, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:47,640 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630222_1099518554852] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 10888 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:57:47,640 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 32302 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 505052, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:47,640 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:47,640 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:47,640 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:47,640 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:47,640 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:47,698 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 237ms (threshold=20ms); ack: seqno: 138845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:47,775 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 138853 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1407983, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:47,855 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 138861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:48,089 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 228ms (threshold=20ms); ack: seqno: 138869 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5463442, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:48,177 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630222_1099518554852] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 11507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 737866, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:57:48,177 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 35199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 668404, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:48,177 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:48,177 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:48,177 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:48,177 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:48,177 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:48,179 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 138877 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1252653, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:48,252 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 138885 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:48,361 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 138956 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1051658, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:48,513 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 139053 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 981018, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:48,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630222_1099518554852] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 12092 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 774766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:57:48,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 37604 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 554592, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:48,647 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:48,647 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:48,647 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:48,647 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:48,648 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:48,654 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 139060 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914504, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:48,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 169ms (threshold=20ms); ack: seqno: 139068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 938057, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:48,913 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:57:48,933 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 139075 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 889695, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:48,955 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:57:48,984 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 139082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 854728, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:49,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630217_1099518554847] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 39305 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 48386609, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:57:49,008 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 12:57:49,008 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 12:57:49,008 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:57:49,008 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 12:57:49,008 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:57:49,017 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025460628 with entries=78055, filesize=122.36 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025468913 2015-10-16 12:57:49,020 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630223_1099518554853] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:49,060 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#34 average throughput is 34.72 MB/sec, slept 10 time(s) and total slept time is 11478 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 12:57:49,080 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/71c97a8cbb7a48cfb9abc6df2f566e4f as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/71c97a8cbb7a48cfb9abc6df2f566e4f 2015-10-16 12:57:49,087 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:57:49,089 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:57:49,099 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b17d9013d33a4b3ca0e5d11e7777c1c0, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b17d9013d33a4b3ca0e5d11e7777c1c0 2015-10-16 12:57:49,107 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/56c38b9cfa9941139a45c7108bf7a330, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/56c38b9cfa9941139a45c7108bf7a330 2015-10-16 12:57:49,116 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a7521413dc0444ce9a26ef76febb7aee, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a7521413dc0444ce9a26ef76febb7aee 2015-10-16 12:57:49,116 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 71c97a8cbb7a48cfb9abc6df2f566e4f(size=778.9 M), total size for store is 11.0 G. This selection was in queue for 0sec, and took 21sec to execute. 2015-10-16 12:57:49,116 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=779.0 M, priority=0, time=36903356074582154; duration=21sec 2015-10-16 12:57:49,116 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:57:49,116 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 6 compacting, 3 eligible, 10 blocking 2015-10-16 12:57:49,117 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:57:49,117 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:57:49,117 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:57:49,238 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 249ms (threshold=20ms); ack: seqno: 139090 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1407269, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:49,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630223_1099518554853] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 684369, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:49,315 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 139098 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013377, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:49,357 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630223_1099518554853] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 1081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 409519, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:49,357 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:57:49,357 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:57:49,357 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:57:49,357 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:57:49,540 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 220ms (threshold=20ms); ack: seqno: 139107 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1182336, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:49,547 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=327356016 > lower bound=16777216 2015-10-16 12:57:49,548 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=312.17 MB 2015-10-16 12:57:49,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630223_1099518554853] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 2288 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 632053, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:49,668 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 139114 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1136101, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:49,880 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 206ms (threshold=20ms); ack: seqno: 139123 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1197721, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:49,959 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 139131 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1243537, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:50,075 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 139139 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1256678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:50,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630223_1099518554853] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 5251 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 547225, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:50,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630225_1099518554855] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 654 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 622834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:57:50,171 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:50,171 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:50,171 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:50,171 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:57:50,171 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:50,276 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 198ms (threshold=20ms); ack: seqno: 139146 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1074062, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:50,352 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 139152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1052877, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:50,463 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 139160 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1063989, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:50,648 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 181ms (threshold=20ms); ack: seqno: 139167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937441, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:50,725 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 139174 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070136, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:50,842 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:50,842 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:50,842 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630225_1099518554855] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 1631 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 646217, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:57:50,842 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630223_1099518554853] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 8822 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 442470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:50,842 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:50,842 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:50,843 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 139181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1056955, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:51,032 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 184ms (threshold=20ms); ack: seqno: 139189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1206682, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:51,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 139196 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1087982, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:51,240 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 139204 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995541, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:51,391 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 147ms (threshold=20ms); ack: seqno: 139211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072786, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:51,524 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630226_1099518554856] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 2621 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 737448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:57:51,524 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630223_1099518554853] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 12361 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 425240, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:51,524 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:57:51,524 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:57:51,524 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:57:51,524 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:57:51,524 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:57:51,559 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 163ms (threshold=20ms); ack: seqno: 139220 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010379, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:51,649 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=7524481, memsize=312.2 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1e574357f2dd474ab7b7b1cfd801f3ca 2015-10-16 12:57:51,654 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1e574357f2dd474ab7b7b1cfd801f3ca as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1e574357f2dd474ab7b7b1cfd801f3ca 2015-10-16 12:57:51,660 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1e574357f2dd474ab7b7b1cfd801f3ca, entries=1136780, sequenceid=7524481, filesize=172.7 M 2015-10-16 12:57:51,662 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~312.22 MB/327382960, currentsize=59.72 MB/62617680 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2115ms, sequenceid=7524481, compaction requested=true 2015-10-16 12:57:51,662 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:57:51,662 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 6 compacting, 4 eligible, 10 blocking 2015-10-16 12:57:51,664 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio 2015-10-16 12:57:51,664 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:57:51,664 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:57:51,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 146ms (threshold=20ms); ack: seqno: 139228 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043091, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:51,888 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 169ms (threshold=20ms); ack: seqno: 139236 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 805104, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:51,974 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 139244 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:52,051 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 139252 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989361, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:52,294 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 238ms (threshold=20ms); ack: seqno: 139260 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1177316, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:52,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630223_1099518554853] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 17266 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 495664, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:52,405 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:52,405 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:52,405 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:52,405 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:57:52,405 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:52,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 139268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1203656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:52,663 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 252ms (threshold=20ms); ack: seqno: 139276 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1171621, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:52,732 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 139283 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1246733, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:52,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 139291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 944197, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:53,036 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 239ms (threshold=20ms); ack: seqno: 139299 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1326115, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:53,094 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 139306 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1112273, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:53,197 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 139314 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1265229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:53,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630223_1099518554853] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 21327 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 759118, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:53,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 206ms (threshold=20ms); ack: seqno: 139321 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1107223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:53,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630223_1099518554853] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 22611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 519772, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:53,457 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:53,457 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:53,457 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:53,457 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:53,457 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:53,484 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 139328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1220560, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:53,560 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 139337 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1028886, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:53,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 281ms (threshold=20ms); ack: seqno: 139344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1254690, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:53,922 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 139352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1243918, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:53,934 DEBUG [PriorityRpcServer.handler=3,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:57:53,934 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134275600 > lower bound=16777216 2015-10-16 12:57:53,934 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:57:53,961 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630223_1099518554853] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 25664 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 604326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:54,183 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 255ms (threshold=20ms); ack: seqno: 139360 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1349231, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:54,274 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:54,274 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630223_1099518554853] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 27128 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 627774, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:54,274 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630227_1099518554857] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 432 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896902, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:57:54,274 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:54,274 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:54,274 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:54,274 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:54,276 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 139368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1201331, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:54,511 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 230ms (threshold=20ms); ack: seqno: 139376 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1346437, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:54,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 139387 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1341436, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:54,683 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 139395 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2774016, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:54,732 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=7571113, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/41c77e4939fa46fb9e167ef092ff2290 2015-10-16 12:57:54,738 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/41c77e4939fa46fb9e167ef092ff2290 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/41c77e4939fa46fb9e167ef092ff2290 2015-10-16 12:57:54,743 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/41c77e4939fa46fb9e167ef092ff2290, entries=466280, sequenceid=7571113, filesize=70.8 M 2015-10-16 12:57:54,745 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134284400, currentsize=22.38 MB/23468720 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 811ms, sequenceid=7571113, compaction requested=true 2015-10-16 12:57:54,746 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:57:54,746 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 6 compacting, 5 eligible, 10 blocking 2015-10-16 12:57:54,748 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 6 permutations with 0 in ratio 2015-10-16 12:57:54,748 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:57:54,748 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:57:54,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 214ms (threshold=20ms); ack: seqno: 139402 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1514005, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:55,021 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630223_1099518554853] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 31106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 448873, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:55,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 139410 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:55,066 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 85ms (threshold=20ms) 2015-10-16 12:57:55,066 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 86ms (threshold=20ms) 2015-10-16 12:57:55,066 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 86ms (threshold=20ms) 2015-10-16 12:57:55,066 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:57:55,066 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 86ms (threshold=20ms) 2015-10-16 12:57:55,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 161ms (threshold=20ms); ack: seqno: 139417 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992590, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:55,312 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 139424 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 959584, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:55,398 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 139432 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1183179, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:55,666 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 264ms (threshold=20ms); ack: seqno: 139439 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1267109, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:55,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 139447 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1500508, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:55,978 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 239ms (threshold=20ms); ack: seqno: 139454 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1343034, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:56,021 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630223_1099518554853] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 36531 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 640609, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:56,021 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 139460 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1360701, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:56,021 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:56,021 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:56,021 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:56,021 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:56,021 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:56,140 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 139462 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1250472, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:56,189 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 139466 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1226114, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:56,342 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 139561 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909583, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:56,423 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 139568 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 949882, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:56,433 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:57:56,556 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 139575 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089652, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:56,628 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 139584 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 877495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:56,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 139597 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1065598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:56,811 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 139604 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975051, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:57,036 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#28 7757397/7547497 (102.78%), rate=11400.98 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=unlimited, activeCompactions=1] 2015-10-16 12:57:57,039 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 221ms (threshold=20ms); ack: seqno: 139609 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1488420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:57,040 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630223_1099518554853] hdfs.DFSClient: Slow ReadProcessor read fields took 565ms (threshold=20ms); ack: seqno: 39216 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2214428, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:57:57,041 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 565ms (threshold=20ms) 2015-10-16 12:57:57,050 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025468913 with entries=78064, filesize=122.37 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025476433 2015-10-16 12:57:57,054 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630228_1099518554858] hdfs.DFSClient: Slow ReadProcessor read fields took 599ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1149055, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:57:57,055 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025435250 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025435250 2015-10-16 12:57:57,066 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025460628 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025460628 2015-10-16 12:57:57,090 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 139615 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1213644, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:57,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630218_1099518554848] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 139621 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 778800, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:57,493 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630228_1099518554858] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 2092 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 428978, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:57:57,493 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 101ms (threshold=20ms) 2015-10-16 12:57:57,493 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 101ms (threshold=20ms) 2015-10-16 12:57:57,493 INFO [sync.3] wal.FSHLog: Slow sync cost: 101 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:57:57,493 INFO [sync.1] wal.FSHLog: Slow sync cost: 101 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:57:57,493 INFO [sync.2] wal.FSHLog: Slow sync cost: 101 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:57:57,494 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 102ms (threshold=20ms) 2015-10-16 12:57:57,494 INFO [sync.4] wal.FSHLog: Slow sync cost: 102 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:57:57,494 INFO [sync.0] wal.FSHLog: Slow sync cost: 102 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:57:58,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630228_1099518554858] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 7152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 700390, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:57:58,482 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630228_1099518554858] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 7282 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 377436, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:57:58,482 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:58,482 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:58,482 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:58,482 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:57:58,869 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 139628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1553595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:58,922 DEBUG [PriorityRpcServer.handler=17,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:57:58,922 WARN [MemStoreFlusher.0] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:57:58,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:57:58,922 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 6 compacting, 5 eligible, 10 blocking 2015-10-16 12:57:58,925 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 6 permutations with 0 in ratio 2015-10-16 12:57:58,925 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:57:58,925 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:57:59,022 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 139721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2541314, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:59,146 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 139728 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:59,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 139735 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984866, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:59,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630228_1099518554858] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 12637 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 413641, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:57:59,405 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:59,405 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:59,405 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:57:59,406 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:57:59,406 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 154ms (threshold=20ms); ack: seqno: 139743 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 933815, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:59,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 139751 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917360, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:59,503 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 139760 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819313, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:59,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 139768 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 938468, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:59,657 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 139776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 933369, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:59,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 139783 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812181, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:59,779 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 139788 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941771, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:57:59,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 139799 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887750, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:00,047 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 139806 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1084012, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:00,209 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 146ms (threshold=20ms); ack: seqno: 139821 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020487, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:00,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 139828 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905036, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:00,317 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630228_1099518554858] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 17781 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 410939, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:58:00,317 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:00,317 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:00,317 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:00,317 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:00,317 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:00,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 139843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1080605, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:00,458 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 139854 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066663, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:00,595 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 139869 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 997311, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:00,700 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 139878 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:00,804 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 139886 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 998854, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:00,925 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 139894 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965490, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:00,969 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 139902 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835721, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:01,095 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 139910 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836671, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:01,221 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630228_1099518554858] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 23003 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 465898, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:58:01,221 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:58:01,221 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:58:01,221 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:58:01,221 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 12:58:01,223 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 139922 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1051975, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:01,376 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 139937 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:01,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 207ms (threshold=20ms); ack: seqno: 139944 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070315, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:01,714 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 140029 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859985, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:01,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 140163 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 897421, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:01,997 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630228_1099518554858] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 27181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 493825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:58:01,997 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 140266 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 805847, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:01,997 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:58:01,997 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:58:01,997 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:58:01,997 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:58:01,997 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:58:02,057 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 140274 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 918415, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:02,189 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 140313 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870099, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:02,628 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 140823 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 677902, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:02,628 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630228_1099518554858] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 30451 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 661889, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:58:02,628 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:58:02,628 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:58:02,628 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:58:02,628 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:58:02,628 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:58:02,909 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630228_1099518554858] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 31314 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 730733, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:58:03,241 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:03,241 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630228_1099518554858] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 32926 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 604605, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:58:03,241 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630229_1099518554859] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 141572 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 774253, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:03,241 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:03,241 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:03,241 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:03,241 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:03,799 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630230_1099518554860] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 142245 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 831400, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:03,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630228_1099518554858] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 35734 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 774280, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:58:03,800 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:03,800 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:03,800 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:03,800 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:04,352 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#28 average throughput is 23.18 MB/sec, slept 134 time(s) and total slept time is 88631 ms. 0 active compactions remaining, total limit is unlimited 2015-10-16 12:58:04,395 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630228_1099518554858] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 38672 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 452173, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:58:04,395 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:04,395 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:04,395 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:04,395 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:04,395 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:04,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630230_1099518554860] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 142893 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 916944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:04,416 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/40708f5c7dee4e848af4976f786425f6 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/40708f5c7dee4e848af4976f786425f6 2015-10-16 12:58:04,431 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:58:04,432 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:58:04,440 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c01239c636ca4b4eae6973f4079040fd, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c01239c636ca4b4eae6973f4079040fd 2015-10-16 12:58:04,448 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/81f31066d4f0409798a1d4a9948c5037, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/81f31066d4f0409798a1d4a9948c5037 2015-10-16 12:58:04,456 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fafe4baf306a4939a566fe0606570530, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/fafe4baf306a4939a566fe0606570530 2015-10-16 12:58:04,465 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b5ea6dcde76c40d59c0e884645a259a8, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b5ea6dcde76c40d59c0e884645a259a8 2015-10-16 12:58:04,473 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b393d73d6f5443148f2587609a8e38f9, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b393d73d6f5443148f2587609a8e38f9 2015-10-16 12:58:04,481 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/832e5c4ada5a487599575c458b72ce33, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/832e5c4ada5a487599575c458b72ce33 2015-10-16 12:58:04,482 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 6 (all) file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 40708f5c7dee4e848af4976f786425f6(size=8.6 G), total size for store is 11.2 G. This selection was in queue for 0sec, and took 6mins, 8sec to execute. 2015-10-16 12:58:04,482 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=6, fileSize=8.6 G, priority=4, time=36903024948261836; duration=6mins, 8sec 2015-10-16 12:58:04,482 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:58:04,490 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:58:04,524 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025476433 with entries=77838, filesize=122.02 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025484490 2015-10-16 12:58:05,222 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=323997200 > lower bound=16777216 2015-10-16 12:58:05,222 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=308.93 MB 2015-10-16 12:58:05,317 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630231_1099518554861] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 4084 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 549496, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:05,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630232_1099518554862] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 122 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834366, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:05,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630231_1099518554861] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 4553 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 575516, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:05,457 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:58:05,457 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:58:05,457 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:58:05,457 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:58:05,457 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:58:06,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630231_1099518554861] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 8091 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 493596, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:06,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630232_1099518554862] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 1195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804778, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:06,164 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:06,164 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:06,164 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:06,164 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:06,164 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:06,861 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630233_1099518554863] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 2219 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006622, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:06,861 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630231_1099518554861] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 11636 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 488545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:06,861 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:06,861 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:06,861 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:06,861 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:07,265 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=7683946, memsize=309.9 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/0c5503585a8c4223b46a1227560c0951 2015-10-16 12:58:07,270 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/0c5503585a8c4223b46a1227560c0951 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0c5503585a8c4223b46a1227560c0951 2015-10-16 12:58:07,276 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0c5503585a8c4223b46a1227560c0951, entries=1128280, sequenceid=7683946, filesize=171.4 M 2015-10-16 12:58:07,277 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~309.88 MB/324934720, currentsize=54.13 MB/56760320 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2055ms, sequenceid=7683946, compaction requested=true 2015-10-16 12:58:07,278 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:58:07,278 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 0 compacting, 7 eligible, 10 blocking 2015-10-16 12:58:07,280 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 509341413 starting at candidate #3 after considering 15 permutations with 2 in ratio 2015-10-16 12:58:07,281 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:58:07,281 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:58:07,281 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=485.7 M 2015-10-16 12:58:07,281 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4440a49a5fa342439b2ef0dd7454b02b, keycount=466310, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=7410798 2015-10-16 12:58:07,281 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1e574357f2dd474ab7b7b1cfd801f3ca, keycount=1136780, bloomtype=ROW, size=172.7 M, encoding=NONE, seqNum=7524481 2015-10-16 12:58:07,281 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/41c77e4939fa46fb9e167ef092ff2290, keycount=466280, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=7571113 2015-10-16 12:58:07,281 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0c5503585a8c4223b46a1227560c0951, keycount=1128280, bloomtype=ROW, size=171.4 M, encoding=NONE, seqNum=7683946 2015-10-16 12:58:07,306 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:58:07,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630231_1099518554861] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 14941 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 24228512, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:07,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630234_1099518554864] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 242 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 24885475, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:58:07,555 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 59ms (threshold=20ms) 2015-10-16 12:58:07,555 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 59ms (threshold=20ms) 2015-10-16 12:58:07,555 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 61ms (threshold=20ms) 2015-10-16 12:58:07,555 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 61ms (threshold=20ms) 2015-10-16 12:58:07,555 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 61ms (threshold=20ms) 2015-10-16 12:58:08,117 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630231_1099518554861] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 17509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 487433, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:08,117 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630234_1099518554864] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 992 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 688974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:58:08,118 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:08,118 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:08,118 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:08,118 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:08,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630234_1099518554864] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 1724 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 644915, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:58:08,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630231_1099518554861] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 20054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 487490, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:08,647 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:08,647 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:08,647 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:08,647 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:08,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630231_1099518554861] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 20822 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 796548, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:09,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630235_1099518554865] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 2548 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 695715, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:58:09,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630231_1099518554861] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 22331 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 666008, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:09,235 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 12:58:09,235 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:58:09,235 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:58:09,235 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:58:09,235 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:58:09,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630231_1099518554861] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 24905 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 593454, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:09,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630235_1099518554865] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 3270 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 663601, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:58:09,765 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:09,765 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:09,765 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:09,765 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:09,765 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:10,270 DEBUG [PriorityRpcServer.handler=17,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:58:10,270 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134262256 > lower bound=16777216 2015-10-16 12:58:10,290 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.51 MB 2015-10-16 12:58:10,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630235_1099518554865] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 3974 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 703307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:58:10,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630231_1099518554861] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 27560 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864057, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:10,796 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630231_1099518554861] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 29523 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 600228, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:10,796 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630237_1099518554867] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 4546 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 752205, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:58:10,796 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630236_1099518554866] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 590 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896125, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:10,796 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:10,796 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:10,796 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:10,796 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:11,173 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=7730760, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/cbce6832938342ad97e6c43061301946 2015-10-16 12:58:11,178 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/cbce6832938342ad97e6c43061301946 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cbce6832938342ad97e6c43061301946 2015-10-16 12:58:11,184 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cbce6832938342ad97e6c43061301946, entries=468100, sequenceid=7730760, filesize=71.1 M 2015-10-16 12:58:11,187 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.56 MB/134809040, currentsize=21.44 MB/22480800 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 897ms, sequenceid=7730760, compaction requested=true 2015-10-16 12:58:11,187 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:58:11,187 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 4 compacting, 1 eligible, 10 blocking 2015-10-16 12:58:11,188 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:58:11,188 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:58:11,188 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:58:11,263 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630231_1099518554861] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 31566 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 735224, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:11,263 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630237_1099518554867] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 5125 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 877228, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:58:11,263 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:11,263 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:11,263 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:11,263 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:11,263 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:11,824 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630237_1099518554867] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 5844 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 892929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:58:11,825 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630231_1099518554861] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 34209 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 597166, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:11,825 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:11,825 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:11,825 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:11,825 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:11,825 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:12,537 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630231_1099518554861] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 37755 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 395214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:12,537 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:12,537 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:12,537 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:12,537 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:12,537 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:12,561 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630237_1099518554867] hdfs.DFSClient: Slow ReadProcessor read fields took 415ms (threshold=20ms); ack: seqno: 6250 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1000639, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:58:12,746 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:58:12,781 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025484490 with entries=77806, filesize=121.97 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025492746 2015-10-16 12:58:12,783 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025468913 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025468913 2015-10-16 12:58:12,790 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025476433 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025476433 2015-10-16 12:58:13,121 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630238_1099518554868] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 7003 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 715986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:58:13,121 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 1587 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 586362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:13,121 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:13,121 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:13,121 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:13,121 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:13,121 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:13,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 2936 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 75898613, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:13,447 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 76ms (threshold=20ms) 2015-10-16 12:58:13,447 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 76ms (threshold=20ms) 2015-10-16 12:58:13,447 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 76ms (threshold=20ms) 2015-10-16 12:58:13,447 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 76ms (threshold=20ms) 2015-10-16 12:58:13,447 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 77ms (threshold=20ms) 2015-10-16 12:58:13,698 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 4108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 430344, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:13,698 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630238_1099518554868] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 7724 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 737849, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:58:13,698 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:13,698 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:13,698 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:13,698 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:13,698 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:13,831 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#35 average throughput is 72.48 MB/sec, slept 0 time(s) and total slept time is 0 ms. 0 active compactions remaining, total limit is unlimited 2015-10-16 12:58:13,853 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/008ad9db86cb441b8299e952f6e19a04 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/008ad9db86cb441b8299e952f6e19a04 2015-10-16 12:58:13,861 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:58:13,865 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:58:13,873 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4440a49a5fa342439b2ef0dd7454b02b, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4440a49a5fa342439b2ef0dd7454b02b 2015-10-16 12:58:13,881 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1e574357f2dd474ab7b7b1cfd801f3ca, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1e574357f2dd474ab7b7b1cfd801f3ca 2015-10-16 12:58:13,889 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/41c77e4939fa46fb9e167ef092ff2290, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/41c77e4939fa46fb9e167ef092ff2290 2015-10-16 12:58:13,914 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0c5503585a8c4223b46a1227560c0951, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0c5503585a8c4223b46a1227560c0951 2015-10-16 12:58:13,914 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 008ad9db86cb441b8299e952f6e19a04(size=485.7 M), total size for store is 11.5 G. This selection was in queue for 0sec, and took 6sec to execute. 2015-10-16 12:58:13,915 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=485.7 M, priority=3, time=36903396134676769; duration=6sec 2015-10-16 12:58:13,915 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:58:14,655 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 8969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 494942, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:14,655 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:14,655 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:14,655 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:14,655 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:14,655 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:14,834 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 9283 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928881, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:15,311 DEBUG [PriorityRpcServer.handler=17,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:58:15,311 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134247136 > lower bound=16777216 2015-10-16 12:58:15,333 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.71 MB 2015-10-16 12:58:15,365 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 12054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 578603, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:15,716 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 13599 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 521061, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:15,716 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630240_1099518554870] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 498 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924951, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:58:15,716 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:15,716 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:15,716 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:15,716 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:15,716 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:16,131 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=7777652, memsize=128.8 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/94019a24eb154262b7eab1645e409d30 2015-10-16 12:58:16,136 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/94019a24eb154262b7eab1645e409d30 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/94019a24eb154262b7eab1645e409d30 2015-10-16 12:58:16,142 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/94019a24eb154262b7eab1645e409d30, entries=468870, sequenceid=7777652, filesize=71.2 M 2015-10-16 12:58:16,144 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.77 MB/135030080, currentsize=21.25 MB/22279040 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 811ms, sequenceid=7777652, compaction requested=true 2015-10-16 12:58:16,144 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:58:16,144 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 6 store files, 0 compacting, 6 eligible, 10 blocking 2015-10-16 12:58:16,147 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 5 files of size 3183283624 starting at candidate #1 after considering 10 permutations with 1 in ratio 2015-10-16 12:58:16,147 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:58:16,147 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 6 store files, 0 compacting, 6 eligible, 10 blocking 2015-10-16 12:58:16,149 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 5 files of size 3183283624 starting at candidate #1 after considering 10 permutations with 1 in ratio 2015-10-16 12:58:16,149 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:58:16,149 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:58:16,149 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=3.0 G 2015-10-16 12:58:16,150 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/97dc7feeac8749ef8d85f3ae08fe46a4, keycount=10723290, bloomtype=ROW, size=1.6 G, encoding=NONE, seqNum=6851383 2015-10-16 12:58:16,150 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/71c97a8cbb7a48cfb9abc6df2f566e4f, keycount=5127580, bloomtype=ROW, size=778.9 M, encoding=NONE, seqNum=7364163 2015-10-16 12:58:16,150 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/008ad9db86cb441b8299e952f6e19a04, keycount=3197650, bloomtype=ROW, size=485.7 M, encoding=NONE, seqNum=7683946 2015-10-16 12:58:16,150 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cbce6832938342ad97e6c43061301946, keycount=468100, bloomtype=ROW, size=71.1 M, encoding=NONE, seqNum=7730760 2015-10-16 12:58:16,150 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/94019a24eb154262b7eab1645e409d30, keycount=468870, bloomtype=ROW, size=71.2 M, encoding=NONE, seqNum=7777652 2015-10-16 12:58:16,191 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:58:16,383 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630241_1099518554871] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 702214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:58:16,383 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 16875 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 454518, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:16,383 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:16,383 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:16,383 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:16,383 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:16,383 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:16,820 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630241_1099518554871] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 792 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1065600, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:58:16,939 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 19608 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 629796, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:16,939 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630241_1099518554871] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 898 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861826, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:58:16,939 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:16,939 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:16,939 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:16,939 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:16,939 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:17,483 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 22290 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701071, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:17,483 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630241_1099518554871] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 1612 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 716616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:58:17,483 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:17,483 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:17,483 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:17,483 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:17,483 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:17,965 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630241_1099518554871] hdfs.DFSClient: Slow ReadProcessor read fields took 162ms (threshold=20ms); ack: seqno: 2044 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:58:18,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 25573 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 469629, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:18,133 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:18,133 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:18,133 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:18,133 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:18,133 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:18,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 2169 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1250158, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:18,191 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 2177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1260010, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:18,351 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 2329 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020057, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:18,523 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 2465 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1233488, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:18,695 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 2607 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1131614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:18,812 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 28921 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 633826, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:18,812 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 2703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885131, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:18,812 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:18,812 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:18,812 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:18,869 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 2707 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1133728, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:19,050 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 2841 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1231634, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:19,229 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 2973 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1109693, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:19,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 3117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069756, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:19,504 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 32479 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 568134, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:19,504 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 3204 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 725853, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:19,504 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:19,504 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:19,504 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:19,504 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:19,504 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:19,572 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 3211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844369, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:19,759 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 3351 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021977, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:19,964 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 3492 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:20,026 DEBUG [PriorityRpcServer.handler=18,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:58:20,026 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134268352 > lower bound=16777216 2015-10-16 12:58:20,027 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:58:20,055 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 35328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917983, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:20,161 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 3641 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 829384, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:20,249 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630243_1099518554873] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 209 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 725655, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:58:20,251 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 157ms (threshold=20ms); ack: seqno: 35534 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857416, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:20,323 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 3721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947554, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:20,509 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 3860 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 810077, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:20,789 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:20,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630242_1099518554872] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 4064 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927154, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:20,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630239_1099518554869] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 38120 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 764045, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:20,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630243_1099518554873] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 1006 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 733728, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:58:20,789 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:20,789 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:20,789 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:20,790 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:20,897 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=7824281, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ff888a3559ed4df2ae17816179e042bd 2015-10-16 12:58:20,902 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ff888a3559ed4df2ae17816179e042bd as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ff888a3559ed4df2ae17816179e042bd 2015-10-16 12:58:20,918 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ff888a3559ed4df2ae17816179e042bd, entries=466250, sequenceid=7824281, filesize=70.8 M 2015-10-16 12:58:20,919 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134276000, currentsize=19.01 MB/19937760 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 893ms, sequenceid=7824281, compaction requested=false 2015-10-16 12:58:20,927 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:58:20,973 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025492746 with entries=77875, filesize=122.07 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025500928 2015-10-16 12:58:20,977 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010564, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:20,978 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025484490 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025484490 2015-10-16 12:58:21,353 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 1741 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 468174, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:21,353 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630244_1099518554874] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 4719 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728541, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:58:21,353 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:21,354 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:21,353 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:21,353 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:21,354 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:21,905 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630244_1099518554874] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 5400 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 680688, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:58:21,905 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 4481 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 589700, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:21,905 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:58:21,905 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:58:21,905 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:58:21,905 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:58:21,905 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 12:58:22,461 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:58:22,461 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 12:58:22,461 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 12:58:22,461 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 12:58:22,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630244_1099518554874] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 6106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845026, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:58:22,461 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 12:58:22,559 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630244_1099518554874] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 6113 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066032, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:58:22,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630246_1099518554876] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 6257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1391877, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:22,903 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630246_1099518554876] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 6396 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 784318, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:23,176 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 11007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 493173, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:23,176 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:23,176 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:23,176 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:23,176 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:23,176 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:23,178 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630246_1099518554876] hdfs.DFSClient: Slow ReadProcessor read fields took 172ms (threshold=20ms); ack: seqno: 6537 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 846474, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:23,379 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630246_1099518554876] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 6672 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 957269, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:23,650 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630246_1099518554876] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 6955 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 867847, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:23,825 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 14245 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 428814, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:23,825 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:58:23,825 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:23,827 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630246_1099518554876] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 7099 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885086, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:23,901 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630246_1099518554876] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 7106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:24,081 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630246_1099518554876] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 7250 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 799367, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:24,253 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630246_1099518554876] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 7392 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 821853, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:24,424 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630246_1099518554876] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 7532 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 851068, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:24,526 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:24,526 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:24,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630246_1099518554876] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 7623 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 795027, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:24,526 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:24,526 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:24,526 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:24,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630246_1099518554876] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 7631 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790619, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:24,925 DEBUG [PriorityRpcServer.handler=11,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:58:24,926 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134264656 > lower bound=16777216 2015-10-16 12:58:24,926 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:58:24,956 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 19864 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 521526, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:25,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630247_1099518554877] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 150 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 880500, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:58:25,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630246_1099518554876] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 8255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 850053, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:25,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 20397 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 478203, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:25,111 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:25,111 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:25,111 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:25,515 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630248_1099518554878] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 8783 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 24111966, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:25,536 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 21812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 846078, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:25,622 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630247_1099518554877] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 804 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767741, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:58:25,622 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630248_1099518554878] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 8912 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 776432, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:25,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 21992 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 775520, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:25,880 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=7870910, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/cc99f36a775444d9952dc958a62f00ce 2015-10-16 12:58:25,886 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/cc99f36a775444d9952dc958a62f00ce as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cc99f36a775444d9952dc958a62f00ce 2015-10-16 12:58:25,901 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cc99f36a775444d9952dc958a62f00ce, entries=466250, sequenceid=7870910, filesize=70.8 M 2015-10-16 12:58:25,902 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134275760, currentsize=19.29 MB/20228720 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 976ms, sequenceid=7870910, compaction requested=true 2015-10-16 12:58:25,902 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:58:25,903 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 5 compacting, 2 eligible, 10 blocking 2015-10-16 12:58:25,903 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:58:25,903 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:58:25,903 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:58:26,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630248_1099518554878] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 9460 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1056955, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:26,136 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630248_1099518554878] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 9506 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5795216, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:26,136 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 24346 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 15388733, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:26,136 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:26,136 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:26,136 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:26,136 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:26,136 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:26,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 27066 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 481199, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:26,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630248_1099518554878] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 10214 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 694265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 12:58:26,700 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:26,700 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:26,700 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:26,700 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:26,700 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:27,259 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630249_1099518554879] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 10922 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023118, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:27,259 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 29721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 453122, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:27,259 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:27,259 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:27,259 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:27,259 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:27,259 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:27,838 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 32395 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 598941, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:27,838 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630249_1099518554879] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 11636 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 687218, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:27,838 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:27,838 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:27,838 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:27,838 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:27,838 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:28,323 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 34442 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 724902, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:28,323 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630249_1099518554879] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 12142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 793248, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:28,323 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 84ms (threshold=20ms) 2015-10-16 12:58:28,323 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 84ms (threshold=20ms) 2015-10-16 12:58:28,323 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 84ms (threshold=20ms) 2015-10-16 12:58:28,323 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 84ms (threshold=20ms) 2015-10-16 12:58:28,323 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 84ms (threshold=20ms) 2015-10-16 12:58:28,499 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630249_1099518554879] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 12330 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 855169, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:28,499 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 35128 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 605950, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:28,499 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:28,499 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:28,499 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:28,499 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:28,499 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:28,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630250_1099518554880] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 12604 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966268, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:58:29,082 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630245_1099518554875] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 38044 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 547812, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:29,082 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:29,082 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:29,082 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:29,082 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:29,082 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:29,083 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630250_1099518554880] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 12971 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 939349, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:58:29,254 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:58:29,297 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025500928 with entries=77932, filesize=122.17 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025509254 2015-10-16 12:58:29,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1141430, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:29,301 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025492746 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025492746 2015-10-16 12:58:29,657 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630250_1099518554880] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 13628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728164, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:58:29,657 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 1718 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 433615, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:29,657 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:29,657 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:30,098 DEBUG [PriorityRpcServer.handler=6,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:58:30,099 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134273696 > lower bound=16777216 2015-10-16 12:58:30,099 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:58:30,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 4064 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 708013, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:30,237 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630252_1099518554882] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 91 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844778, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:58:30,237 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 4382 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 531576, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:30,237 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630250_1099518554880] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 14288 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863412, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:58:30,237 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:30,237 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:30,237 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:30,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 6476 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 551296, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:30,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630252_1099518554882] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 722 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 775719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:58:30,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630253_1099518554883] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 14836 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874684, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:58:30,689 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:30,689 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:30,689 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:30,689 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:30,689 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:30,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 6579 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 723663, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:30,988 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=7917539, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/9ec025b2ff18414d8c4e09f51a2ed926 2015-10-16 12:58:30,994 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/9ec025b2ff18414d8c4e09f51a2ed926 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9ec025b2ff18414d8c4e09f51a2ed926 2015-10-16 12:58:31,000 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9ec025b2ff18414d8c4e09f51a2ed926, entries=466250, sequenceid=7917539, filesize=70.8 M 2015-10-16 12:58:31,002 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134276160, currentsize=18.64 MB/19545600 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 903ms, sequenceid=7917539, compaction requested=true 2015-10-16 12:58:31,002 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:58:31,002 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 5 compacting, 3 eligible, 10 blocking 2015-10-16 12:58:31,003 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222818123 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 12:58:31,003 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:58:31,004 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:58:31,004 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 12:58:31,004 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ff888a3559ed4df2ae17816179e042bd, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=7824281 2015-10-16 12:58:31,004 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cc99f36a775444d9952dc958a62f00ce, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=7870910 2015-10-16 12:58:31,004 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9ec025b2ff18414d8c4e09f51a2ed926, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=7917539 2015-10-16 12:58:31,018 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:58:31,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630254_1099518554884] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 150 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 698187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:58:31,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 8059 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 515994, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:31,171 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:31,171 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:31,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630253_1099518554883] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 15417 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 823878, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:58:31,171 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:58:31,171 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:58:31,171 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:31,597 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 10108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 451461, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:31,597 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630253_1099518554883] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 15835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 784277, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:58:31,598 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:31,598 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:31,598 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:31,598 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:31,598 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:31,599 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630254_1099518554884] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 578 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873116, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:58:31,987 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630253_1099518554883] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 16280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 802333, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 12:58:31,987 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 11894 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 498034, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:31,987 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630254_1099518554884] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 1059 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 733317, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:58:31,987 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:31,987 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:31,987 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:31,987 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:31,987 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:32,364 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630254_1099518554884] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 1530 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 673074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:58:32,364 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630255_1099518554885] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 16747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1032889, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:58:32,364 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 13558 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 656364, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:32,365 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:32,365 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:32,365 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:32,365 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:32,365 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:32,748 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630254_1099518554884] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 1991 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 712127, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:58:32,748 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630255_1099518554885] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 17212 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 753602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:58:32,748 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 15317 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 478344, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:32,748 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:32,748 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:32,748 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:32,748 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:32,748 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:33,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 16995 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 611527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:33,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630255_1099518554885] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 17694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 687414, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:58:33,128 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:33,128 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:33,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630256_1099518554886] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 2444 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988104, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:58:33,129 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:33,514 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 18674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 494791, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:33,514 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630256_1099518554886] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 2900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758089, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:58:33,514 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630255_1099518554885] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 18186 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 747719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:58:33,514 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:33,514 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:33,514 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:33,514 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:33,514 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:33,848 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630255_1099518554885] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 18612 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 944511, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:58:33,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630255_1099518554885] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 18640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827589, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:58:33,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630256_1099518554886] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 3363 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 816207, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 12:58:33,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 20460 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 563558, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:33,906 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:33,906 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:33,906 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:33,907 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:33,972 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#37 average throughput is 70.03 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 12:58:33,985 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/27e80cf2feb64dd2a48ee3c9f2107b53 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/27e80cf2feb64dd2a48ee3c9f2107b53 2015-10-16 12:58:33,994 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:58:33,998 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:58:34,005 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ff888a3559ed4df2ae17816179e042bd, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ff888a3559ed4df2ae17816179e042bd 2015-10-16 12:58:34,013 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cc99f36a775444d9952dc958a62f00ce, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cc99f36a775444d9952dc958a62f00ce 2015-10-16 12:58:34,021 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9ec025b2ff18414d8c4e09f51a2ed926, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/9ec025b2ff18414d8c4e09f51a2ed926 2015-10-16 12:58:34,022 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 27e80cf2feb64dd2a48ee3c9f2107b53(size=212.5 M), total size for store is 11.7 G. This selection was in queue for 0sec, and took 3sec to execute. 2015-10-16 12:58:34,022 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=2, time=36903419857637975; duration=3sec 2015-10-16 12:58:34,022 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:58:34,429 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 22895 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 411046, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:34,429 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630257_1099518554887] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 19309 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 820413, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:34,429 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:34,429 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:34,429 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:34,970 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 25678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 510271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:34,970 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630257_1099518554887] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 19981 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 757678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:34,970 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:34,970 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:34,970 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:34,970 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:34,970 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:35,273 DEBUG [PriorityRpcServer.handler=10,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:58:35,273 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134261536 > lower bound=16777216 2015-10-16 12:58:35,273 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 12:58:35,298 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 27341 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 469762, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:35,482 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630257_1099518554887] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 20620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869830, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:35,482 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630258_1099518554888] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 203 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842853, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:58:35,482 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 28097 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 599550, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:35,482 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:35,482 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:35,482 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:35,482 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:35,482 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:35,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 29431 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 694718, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:35,998 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 29936 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 547836, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:35,998 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630258_1099518554888] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 958 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 756607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:58:35,998 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630259_1099518554889] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 21185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792509, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:58:35,998 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:58:35,998 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 12:58:35,999 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:36,138 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=7964165, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/6c46eaac11194a5d901446ceeb02da2c 2015-10-16 12:58:36,143 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/6c46eaac11194a5d901446ceeb02da2c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6c46eaac11194a5d901446ceeb02da2c 2015-10-16 12:58:36,149 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6c46eaac11194a5d901446ceeb02da2c, entries=466210, sequenceid=7964165, filesize=70.8 M 2015-10-16 12:58:36,151 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.04 MB/134264000, currentsize=18.90 MB/19813840 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 878ms, sequenceid=7964165, compaction requested=true 2015-10-16 12:58:36,151 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:58:36,151 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 5 compacting, 2 eligible, 10 blocking 2015-10-16 12:58:36,152 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:58:36,152 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:58:36,152 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:58:36,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 32736 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 595073, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:36,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630259_1099518554889] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 21736 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 852455, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:58:36,565 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:36,565 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:36,565 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:36,565 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:36,565 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:36,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630259_1099518554889] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 22047 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863507, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:58:37,157 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 35667 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 524708, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:37,157 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630259_1099518554889] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 22385 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 807349, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:58:37,157 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:37,157 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:37,157 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:37,157 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:37,157 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:37,610 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 37664 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 73936539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:37,610 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:58:37,610 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630259_1099518554889] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 22922 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 46888751, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:58:37,610 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:58:37,610 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:58:37,610 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 75ms (threshold=20ms) 2015-10-16 12:58:37,610 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 74ms (threshold=20ms) 2015-10-16 12:58:37,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630260_1099518554890] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 23131 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 781223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:37,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630251_1099518554881] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 38163 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 511915, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 12:58:37,745 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:37,745 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:37,745 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:37,745 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:37,875 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:58:37,914 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025509254 with entries=77927, filesize=122.15 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025517875 2015-10-16 12:58:37,915 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025500928 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025500928 2015-10-16 12:58:37,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630261_1099518554891] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:38,300 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:38,300 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:38,300 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:38,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630260_1099518554890] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 23815 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 662990, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:38,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630261_1099518554891] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 1900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 383722, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:38,301 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:38,301 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:38,824 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630261_1099518554891] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 4561 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 532259, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:38,825 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630260_1099518554890] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 24503 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 724194, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:38,825 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:38,825 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:38,825 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:38,825 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.7142857142857143, tune compaction throughput to 17.14 MB/sec 2015-10-16 12:58:39,328 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630260_1099518554890] hdfs.DFSClient: Slow ReadProcessor read fields took 427ms (threshold=20ms); ack: seqno: 24609 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1382049, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:39,683 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630261_1099518554891] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 9022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 515341, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:39,684 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:39,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630260_1099518554890] hdfs.DFSClient: Slow ReadProcessor read fields took 447ms (threshold=20ms); ack: seqno: 24776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1349193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:40,154 DEBUG [PriorityRpcServer.handler=15,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:58:40,154 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134265376 > lower bound=16777216 2015-10-16 12:58:40,155 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.03 MB 2015-10-16 12:58:40,182 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630261_1099518554891] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 11747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 576059, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:40,437 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630261_1099518554891] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 12474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 756805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:40,495 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630260_1099518554890] hdfs.DFSClient: Slow ReadProcessor read fields took 463ms (threshold=20ms); ack: seqno: 24943 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070870, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:40,540 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630261_1099518554891] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 12824 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 492830, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:40,540 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630260_1099518554890] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 24951 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062129, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:40,540 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:40,540 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:40,540 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630262_1099518554892] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 722189, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:58:40,540 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:40,540 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:40,540 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:40,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 25093 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1191550, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:40,963 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8010800, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/61acf84e14a34b25a36be496bd277301 2015-10-16 12:58:40,968 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/61acf84e14a34b25a36be496bd277301 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/61acf84e14a34b25a36be496bd277301 2015-10-16 12:58:40,982 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/61acf84e14a34b25a36be496bd277301, entries=466310, sequenceid=8010800, filesize=70.8 M 2015-10-16 12:58:40,985 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134293760, currentsize=18.89 MB/19802400 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 830ms, sequenceid=8010800, compaction requested=true 2015-10-16 12:58:40,985 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:58:40,985 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 5 compacting, 3 eligible, 10 blocking 2015-10-16 12:58:40,986 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:58:40,986 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:58:40,986 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:58:41,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 318ms (threshold=20ms); ack: seqno: 25112 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207452, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:41,294 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630261_1099518554891] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 16428 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 671215, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:41,295 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:41,295 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:41,295 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:41,295 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:41,295 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:41,297 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 214ms (threshold=20ms); ack: seqno: 25119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1396385, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:41,361 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 25126 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1276537, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:41,520 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 25248 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006344, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:41,661 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 25279 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021740, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:41,722 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 25286 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1063676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:41,897 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 25421 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950033, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:42,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630261_1099518554891] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 20584 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 576558, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:42,114 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:42,114 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:42,114 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:42,114 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:42,114 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:42,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 331ms (threshold=20ms); ack: seqno: 25446 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1273559, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:42,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 25452 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1187668, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:42,476 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 25581 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 892230, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:42,827 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 328ms (threshold=20ms); ack: seqno: 25613 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1343097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:42,954 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630261_1099518554891] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 25016 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915798, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:42,954 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 25726 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 634679, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:42,954 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:42,954 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:42,954 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:42,954 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:42,954 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:43,410 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 419ms (threshold=20ms); ack: seqno: 25780 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1145606, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:43,823 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630261_1099518554891] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 29617 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 472548, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:43,823 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:43,823 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:43,823 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:43,993 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 450ms (threshold=20ms); ack: seqno: 25947 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1161424, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:44,576 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 457ms (threshold=20ms); ack: seqno: 26114 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1264642, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:44,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630261_1099518554891] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 33342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 832257, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:44,727 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630261_1099518554891] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 33809 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 494212, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:44,727 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 26274 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022664, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:44,727 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:44,727 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:44,727 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:44,727 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:44,727 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:44,919 DEBUG [PriorityRpcServer.handler=3,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:58:44,919 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134271168 > lower bound=16777216 2015-10-16 12:58:44,919 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:58:44,944 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630261_1099518554891] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 34971 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2240657, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:58:45,159 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 429ms (threshold=20ms); ack: seqno: 26281 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085663, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:45,424 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630264_1099518554894] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 680 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 810341, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:58:45,424 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:45,424 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:45,424 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:58:45,424 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:58:45,424 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:58:45,687 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:58:45,721 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025517875 with entries=77880, filesize=122.08 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025525687 2015-10-16 12:58:45,723 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025509254 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025509254 2015-10-16 12:58:45,729 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8057430, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/2a6ecf411fee48a6beb089dd2cdb7e69 2015-10-16 12:58:45,734 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/2a6ecf411fee48a6beb089dd2cdb7e69 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2a6ecf411fee48a6beb089dd2cdb7e69 2015-10-16 12:58:45,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 466ms (threshold=20ms); ack: seqno: 26448 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1104181, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:45,749 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2a6ecf411fee48a6beb089dd2cdb7e69, entries=466260, sequenceid=8057430, filesize=70.8 M 2015-10-16 12:58:45,751 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134278240, currentsize=22.55 MB/23649440 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 832ms, sequenceid=8057430, compaction requested=true 2015-10-16 12:58:45,751 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:58:45,751 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 5 compacting, 4 eligible, 10 blocking 2015-10-16 12:58:45,753 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 445639663 starting at candidate #0 after considering 3 permutations with 2 in ratio 2015-10-16 12:58:45,753 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:58:45,753 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:58:45,753 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=425.0 M 2015-10-16 12:58:45,753 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/27e80cf2feb64dd2a48ee3c9f2107b53, keycount=1398750, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=7917539 2015-10-16 12:58:45,753 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6c46eaac11194a5d901446ceeb02da2c, keycount=466210, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=7964165 2015-10-16 12:58:45,753 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/61acf84e14a34b25a36be496bd277301, keycount=466310, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=8010800 2015-10-16 12:58:45,753 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2a6ecf411fee48a6beb089dd2cdb7e69, keycount=466260, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=8057430 2015-10-16 12:58:45,771 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:58:45,853 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 26525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873749, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:46,069 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630265_1099518554895] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 1570 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 469970, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:58:46,069 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:46,069 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:46,069 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:46,069 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:46,908 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 970ms (threshold=20ms); ack: seqno: 26617 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930675, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:46,948 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630266_1099518554896] hdfs.DFSClient: Slow ReadProcessor read fields took 1030ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1298675, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:47,010 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 26705 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914457, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:47,010 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630266_1099518554896] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1322223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:47,010 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630265_1099518554895] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 6712 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043051, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:58:47,011 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:47,011 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:47,011 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:47,011 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:47,011 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:47,137 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 26711 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812596, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:47,901 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630265_1099518554895] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 11454 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 493204, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:58:47,901 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:58:47,901 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:58:47,901 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:58:47,901 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:58:47,901 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:58:48,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630266_1099518554896] hdfs.DFSClient: Slow ReadProcessor read fields took 1005ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1242450, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:48,121 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 922ms (threshold=20ms); ack: seqno: 26784 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 47338486, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:48,389 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 237ms (threshold=20ms); ack: seqno: 26807 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1200244, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:48,662 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 248ms (threshold=20ms); ack: seqno: 26822 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863192, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:48,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630265_1099518554895] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 16145 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 480314, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:58:48,778 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:48,778 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:48,778 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:48,778 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:48,997 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 331ms (threshold=20ms); ack: seqno: 26829 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1297032, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:49,241 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630263_1099518554893] hdfs.DFSClient: Slow ReadProcessor read fields took 159ms (threshold=20ms); ack: seqno: 26951 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986957, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:58:49,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630266_1099518554896] hdfs.DFSClient: Slow ReadProcessor read fields took 1044ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207699, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:49,419 DEBUG [PriorityRpcServer.handler=16,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:58:49,419 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134274864 > lower bound=16777216 2015-10-16 12:58:49,419 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:58:49,445 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630265_1099518554895] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 19550 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 784511, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:58:49,513 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630265_1099518554895] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 19730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 370044, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:58:49,514 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:49,516 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:49,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630265_1099518554895] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 19911 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 735845, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:58:50,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630268_1099518554898] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 1152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 48969872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 12:58:50,245 WARN [MemStoreFlusher.1] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:58:50,254 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8104064, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/65d8ca96dfa84218826aba119767ffd2 2015-10-16 12:58:50,260 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/65d8ca96dfa84218826aba119767ffd2 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/65d8ca96dfa84218826aba119767ffd2 2015-10-16 12:58:50,266 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/65d8ca96dfa84218826aba119767ffd2, entries=466300, sequenceid=8104064, filesize=70.8 M 2015-10-16 12:58:50,267 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134290000, currentsize=19.30 MB/20234400 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 848ms, sequenceid=8104064, compaction requested=false 2015-10-16 12:58:50,312 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630265_1099518554895] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 23136 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 524014, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:58:50,313 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:50,313 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:58:50,313 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 12:58:50,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 1026ms (threshold=20ms); ack: seqno: 27123 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1036753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:50,464 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 27124 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1772717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:50,474 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630266_1099518554896] hdfs.DFSClient: Slow ReadProcessor read fields took 1074ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 28847333, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:50,618 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 27191 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1308108, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:51,123 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630265_1099518554895] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 27354 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 468896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:58:51,123 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:51,123 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:51,123 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:51,573 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 882ms (threshold=20ms); ack: seqno: 27289 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081200, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:51,612 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630266_1099518554896] hdfs.DFSClient: Slow ReadProcessor read fields took 1032ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1296640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:51,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630265_1099518554895] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 31553 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 493407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:58:51,932 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:51,932 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:51,933 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:51,933 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:52,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 1030ms (threshold=20ms); ack: seqno: 27456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1151433, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:52,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630266_1099518554896] hdfs.DFSClient: Slow ReadProcessor read fields took 1036ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1191967, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:52,866 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630265_1099518554895] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 36585 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 542113, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:58:52,866 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:52,866 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630266_1099518554896] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 1045 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 737123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:52,866 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 27530 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 775339, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:52,866 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:52,866 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:52,866 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:53,315 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:58:53,354 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025525687 with entries=77898, filesize=122.11 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025533316 2015-10-16 12:58:53,357 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630269_1099518554899] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1133476, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:53,358 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025517875 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025517875 2015-10-16 12:58:53,761 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:53,761 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630269_1099518554899] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 2072 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 386144, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:53,761 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:53,761 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:53,761 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:53,762 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:53,905 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 936ms (threshold=20ms); ack: seqno: 27623 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1342373, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:53,944 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630266_1099518554896] hdfs.DFSClient: Slow ReadProcessor read fields took 954ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1178078, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:54,061 DEBUG [PriorityRpcServer.handler=18,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:58:54,061 WARN [MemStoreFlusher.0] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:58:54,061 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:58:54,395 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630269_1099518554899] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 4994 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 650878, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:54,649 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630269_1099518554899] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 6211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 513331, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:54,649 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:54,649 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:54,649 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:54,649 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:54,649 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:58:55,071 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 1000ms (threshold=20ms); ack: seqno: 27790 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1343255, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:55,095 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 27791 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2353724, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:55,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630266_1099518554896] hdfs.DFSClient: Slow ReadProcessor read fields took 1007ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1135078, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:55,429 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630269_1099518554899] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 10303 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 457498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:55,429 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:55,429 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:55,429 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:55,429 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:55,429 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:56,237 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 970ms (threshold=20ms); ack: seqno: 27957 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1519804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:56,264 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 27962 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1164706, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:56,276 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630266_1099518554896] hdfs.DFSClient: Slow ReadProcessor read fields took 995ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1186737, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:56,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630269_1099518554899] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 15371 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 582676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:56,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 28031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782421, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:56,372 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:56,372 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:56,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630266_1099518554896] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 1555 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 707106, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:56,372 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:56,373 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:56,373 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:56,439 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 28038 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:57,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630269_1099518554899] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 20132 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 604413, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:57,235 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:57,235 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:57,235 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:57,235 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:57,235 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:57,403 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 891ms (threshold=20ms); ack: seqno: 28124 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1451632, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:57,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630266_1099518554896] hdfs.DFSClient: Slow ReadProcessor read fields took 972ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1241996, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:57,513 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 28130 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1105709, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:57,758 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 141ms (threshold=20ms); ack: seqno: 28234 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006261, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:58,054 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630269_1099518554899] hdfs.DFSClient: Slow ReadProcessor read fields took 152ms (threshold=20ms); ack: seqno: 23760 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 508825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:58,054 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 153ms (threshold=20ms) 2015-10-16 12:58:58,054 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 153ms (threshold=20ms) 2015-10-16 12:58:58,054 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 153ms (threshold=20ms) 2015-10-16 12:58:58,054 INFO [sync.1] wal.FSHLog: Slow sync cost: 152 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:58,054 INFO [sync.3] wal.FSHLog: Slow sync cost: 152 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:58,054 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 153ms (threshold=20ms) 2015-10-16 12:58:58,054 INFO [sync.0] wal.FSHLog: Slow sync cost: 153 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:58,054 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 153ms (threshold=20ms) 2015-10-16 12:58:58,054 INFO [sync.4] wal.FSHLog: Slow sync cost: 152 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:58,054 INFO [sync.2] wal.FSHLog: Slow sync cost: 152 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:58,200 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630269_1099518554899] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 24325 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 435716, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:58,200 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:58,200 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:58,200 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:58,200 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:58:58,570 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 769ms (threshold=20ms); ack: seqno: 28291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1324863, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:58,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630266_1099518554896] hdfs.DFSClient: Slow ReadProcessor read fields took 1019ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1147743, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:58,634 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 28295 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1189983, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:58,824 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 28390 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974449, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:59,040 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630269_1099518554899] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 28285 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 757917, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:59,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630269_1099518554899] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 28508 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 823915, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:59,120 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:59,120 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:58:59,120 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:59,120 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:59,120 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:58:59,736 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 852ms (threshold=20ms); ack: seqno: 28458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1270121, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:58:59,774 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630266_1099518554896] hdfs.DFSClient: Slow ReadProcessor read fields took 1011ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:59,923 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630269_1099518554899] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 32779 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 444549, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:58:59,923 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:59,923 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 2138 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3240632, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:58:59,923 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:58:59,924 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:00,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630269_1099518554899] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 38248 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 483180, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:59:00,913 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:00,913 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:00,913 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:00,913 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:00,913 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:00,915 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 1043ms (threshold=20ms); ack: seqno: 28625 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1474442, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:00,940 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 1000ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1129890, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:01,065 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:59:01,095 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025533316 with entries=77781, filesize=121.93 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025541065 2015-10-16 12:59:01,704 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 3152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 421894, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:01,704 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:59:01,704 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:59:01,704 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:01,704 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:59:01,704 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:59:02,081 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 995ms (threshold=20ms); ack: seqno: 28794 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1178999, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:02,106 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 995ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1340919, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:02,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 7344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 484232, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:02,512 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:02,512 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:59:02,512 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:02,512 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:59:02,512 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:03,247 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 1037ms (threshold=20ms); ack: seqno: 28961 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1147970, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:03,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 1045ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1132760, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:03,293 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 28967 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1001218, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:03,348 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 11420 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 750640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:03,440 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:59:03,440 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 11673 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 456246, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:03,440 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 29112 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 770891, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:03,440 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:59:03,440 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:59:03,441 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:59:03,440 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:59:03,684 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 12602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 76657456, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:03,684 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 77ms (threshold=20ms) 2015-10-16 12:59:03,684 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 77ms (threshold=20ms) 2015-10-16 12:59:03,684 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 77ms (threshold=20ms) 2015-10-16 12:59:03,684 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 76ms (threshold=20ms) 2015-10-16 12:59:03,684 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 77ms (threshold=20ms) 2015-10-16 12:59:04,414 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 962ms (threshold=20ms); ack: seqno: 29127 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1327763, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:04,438 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 1049ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1182879, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:04,507 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 17004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 546045, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:04,507 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 2716 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 745186, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:04,507 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:59:04,507 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:59:04,507 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:59:04,507 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:59:04,507 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:59:04,510 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630267_1099518554897] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 29134 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1264459, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:05,352 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 21430 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 637771, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:05,353 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:05,353 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:05,353 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:05,353 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:05,353 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:59:05,579 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 934ms (threshold=20ms); ack: seqno: 29297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1443787, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:05,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 1007ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1227949, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:05,642 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 29304 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1390017, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:05,839 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 29439 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1009837, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:06,229 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 25691 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 473685, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:06,229 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:06,229 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:06,229 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:06,229 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:06,229 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:06,306 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 25955 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 31002474, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:06,306 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:59:06,306 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:59:06,306 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 12:59:06,307 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:59:06,307 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 12:59:06,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 889ms (threshold=20ms); ack: seqno: 29464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1252390, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:06,771 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 1049ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986098, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:06,863 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 29471 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1107676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:07,124 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 29976 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 469141, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:07,124 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:07,124 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:59:07,124 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:07,124 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:07,124 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 12:59:07,126 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 157ms (threshold=20ms); ack: seqno: 29603 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1173222, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:07,176 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 29610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1009272, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:07,911 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 715ms (threshold=20ms); ack: seqno: 29631 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1510155, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:07,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 1045ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1119744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:08,042 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 29729 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 816435, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:08,042 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 3255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 935615, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:08,042 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 34730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 586988, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:08,042 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:08,043 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:08,043 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:08,043 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:08,043 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 12:59:09,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 976ms (threshold=20ms); ack: seqno: 29800 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1468018, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:09,105 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 996ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1465812, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:10,243 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 1042ms (threshold=20ms); ack: seqno: 29967 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1332755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:10,270 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 1047ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1242692, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:11,409 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 1048ms (threshold=20ms); ack: seqno: 30134 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1292773, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:11,437 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 1058ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1267395, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:11,477 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 30141 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1123587, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:11,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 30297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072541, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:12,575 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 901ms (threshold=20ms); ack: seqno: 30301 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1336198, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:12,603 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 1058ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1178806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:12,670 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 3887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 719442, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:12,791 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 211ms (threshold=20ms); ack: seqno: 30307 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1779815, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:12,956 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 30460 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 10089381, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:13,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 785ms (threshold=20ms); ack: seqno: 30468 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1654477, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:13,770 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630270_1099518554900] hdfs.DFSClient: Slow ReadProcessor read fields took 1017ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1252956, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:14,908 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 1042ms (threshold=20ms); ack: seqno: 30635 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1406371, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:14,935 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 1038ms (threshold=20ms); ack: seqno: 4181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1443428, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:16,074 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 1050ms (threshold=20ms); ack: seqno: 30802 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1532774, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:16,101 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 1040ms (threshold=20ms); ack: seqno: 4348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1299886, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:17,237 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#36 3646791/2797530 (130.36%), rate=31036.16 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=17.14 MB/sec, activeCompactions=2] 2015-10-16 12:59:17,240 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 1052ms (threshold=20ms); ack: seqno: 30968 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1307716, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:17,267 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 1047ms (threshold=20ms); ack: seqno: 4517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1393367, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:17,337 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 4554 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2055539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:17,386 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 143ms (threshold=20ms); ack: seqno: 30976 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1254081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:17,504 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#36 sleep 899 ms because current throughput is 37.45 MB/sec, max allowed is 8.57 MB/sec, already slept 39 time(s) and total slept time is 29427 ms till now. 2015-10-16 12:59:18,406 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630272_1099518554902] hdfs.DFSClient: Slow ReadProcessor read fields took 901ms (threshold=20ms); ack: seqno: 31137 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1539979, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 12:59:18,434 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 1015ms (threshold=20ms); ack: seqno: 4684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1344172, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:19,573 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 1023ms (threshold=20ms); ack: seqno: 31308 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1916434, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:19,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 1039ms (threshold=20ms); ack: seqno: 4851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1259506, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:20,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 1059ms (threshold=20ms); ack: seqno: 31474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1706760, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:20,768 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 1057ms (threshold=20ms); ack: seqno: 5018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1417534, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:21,905 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 1055ms (threshold=20ms); ack: seqno: 31641 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1536253, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:21,933 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 1064ms (threshold=20ms); ack: seqno: 5185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1349072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:21,978 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 31699 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 752239, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:21,978 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 5201 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982927, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:23,070 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 1023ms (threshold=20ms); ack: seqno: 31808 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1594052, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:23,100 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 1029ms (threshold=20ms); ack: seqno: 5352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1512323, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:24,236 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 1061ms (threshold=20ms); ack: seqno: 31975 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1408176, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:24,266 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 1064ms (threshold=20ms); ack: seqno: 5521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1387162, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:24,404 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 32140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1208463, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:25,403 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 998ms (threshold=20ms); ack: seqno: 32142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1458129, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:25,432 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 1065ms (threshold=20ms); ack: seqno: 5688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1375152, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:26,570 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 1049ms (threshold=20ms); ack: seqno: 32309 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3246263, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:26,599 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 1065ms (threshold=20ms); ack: seqno: 5855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1197822, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:26,640 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 32315 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1320420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:26,674 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 5924 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 935403, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:26,674 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 32316 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023296, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:27,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 959ms (threshold=20ms); ack: seqno: 32478 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1616674, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:27,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 1031ms (threshold=20ms); ack: seqno: 6022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1346370, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:27,793 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 32484 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1452703, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:28,902 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 997ms (threshold=20ms); ack: seqno: 32645 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1701634, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:28,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 1061ms (threshold=20ms); ack: seqno: 6189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1212486, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:29,006 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630273_1099518554903] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 6250 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1210510, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 12:59:30,070 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 1033ms (threshold=20ms); ack: seqno: 32812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1742336, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:30,097 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630275_1099518554905] hdfs.DFSClient: Slow ReadProcessor read fields took 1038ms (threshold=20ms); ack: seqno: 6358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1263128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:59:31,236 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 1045ms (threshold=20ms); ack: seqno: 32978 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1626725, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:31,263 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630275_1099518554905] hdfs.DFSClient: Slow ReadProcessor read fields took 1049ms (threshold=20ms); ack: seqno: 6525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1369948, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:59:31,316 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630275_1099518554905] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 6545 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782492, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:59:31,316 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 33044 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 765844, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:32,126 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=749, evicted=0, evictedPerRun=0.0 2015-10-16 12:59:32,411 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 1022ms (threshold=20ms); ack: seqno: 33145 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 10044491, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:32,428 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630275_1099518554905] hdfs.DFSClient: Slow ReadProcessor read fields took 1010ms (threshold=20ms); ack: seqno: 6694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1173953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:59:33,568 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630274_1099518554904] hdfs.DFSClient: Slow ReadProcessor read fields took 1047ms (threshold=20ms); ack: seqno: 33314 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1398167, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:33,596 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630275_1099518554905] hdfs.DFSClient: Slow ReadProcessor read fields took 1044ms (threshold=20ms); ack: seqno: 6861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1395592, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:59:33,631 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#38 average throughput is 8.64 MB/sec, slept 41 time(s) and total slept time is 42228 ms. 1 active compactions remaining, total limit is 17.14 MB/sec 2015-10-16 12:59:33,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630275_1099518554905] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 6919 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1131974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 12:59:33,673 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:33,682 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/48c4646896a846498a1f8fdbb315ae3c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/48c4646896a846498a1f8fdbb315ae3c 2015-10-16 12:59:33,690 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 25568ms (threshold=20ms); ack: seqno: 35130 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 805120, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:33,690 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 12:59:33,694 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 12:59:33,701 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/27e80cf2feb64dd2a48ee3c9f2107b53, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/27e80cf2feb64dd2a48ee3c9f2107b53 2015-10-16 12:59:33,709 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6c46eaac11194a5d901446ceeb02da2c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6c46eaac11194a5d901446ceeb02da2c 2015-10-16 12:59:33,718 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/61acf84e14a34b25a36be496bd277301, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/61acf84e14a34b25a36be496bd277301 2015-10-16 12:59:33,726 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2a6ecf411fee48a6beb089dd2cdb7e69, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2a6ecf411fee48a6beb089dd2cdb7e69 2015-10-16 12:59:33,726 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 48c4646896a846498a1f8fdbb315ae3c(size=425.0 M), total size for store is 12.0 G. This selection was in queue for 0sec, and took 47sec to execute. 2015-10-16 12:59:33,726 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=425.0 M, priority=0, time=36903434606976740; duration=47sec 2015-10-16 12:59:33,726 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:59:33,727 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 5 compacting, 2 eligible, 10 blocking 2015-10-16 12:59:33,728 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 12:59:33,728 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:59:33,728 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:59:34,151 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 467ms (threshold=20ms); ack: seqno: 33485 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1548174, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:34,223 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 33489 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1424473, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:34,580 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536919136 > lower bound=16777216 2015-10-16 12:59:34,580 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 12:59:34,682 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 993ms (threshold=20ms); ack: seqno: 35131 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:34,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 411ms (threshold=20ms); ack: seqno: 33651 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1177611, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:34,897 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 161ms (threshold=20ms); ack: seqno: 33656 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1384433, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:35,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 33792 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1448285, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:35,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630277_1099518554907] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 489 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1808956, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:35,073 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 33801 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:35,316 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 232ms (threshold=20ms); ack: seqno: 33818 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088696, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:35,378 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 33824 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993068, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:35,557 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 33981 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1030224, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:35,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 340ms (threshold=20ms); ack: seqno: 33985 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1267356, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:35,966 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630277_1099518554907] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 2082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26227102, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:36,311 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630278_1099518554908] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 2670 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 730960, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 12:59:36,482 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 471ms (threshold=20ms); ack: seqno: 34152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1192529, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:37,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 477ms (threshold=20ms); ack: seqno: 34319 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1415361, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=617152, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 12:59:37,476 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8290505, memsize=512.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/31dd1a1f411b41608038c59a0ec77c03 2015-10-16 12:59:37,483 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/31dd1a1f411b41608038c59a0ec77c03 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/31dd1a1f411b41608038c59a0ec77c03 2015-10-16 12:59:37,497 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/31dd1a1f411b41608038c59a0ec77c03, entries=1864360, sequenceid=8290505, filesize=283.2 M 2015-10-16 12:59:37,498 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 2816ms (threshold=20ms); ack: seqno: 35132 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811263, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:37,498 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536918720, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2918ms, sequenceid=8290505, compaction requested=true 2015-10-16 12:59:37,498 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:59:37,499 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 5 compacting, 3 eligible, 10 blocking 2015-10-16 12:59:37,500 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 12:59:37,500 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 12:59:37,500 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 12:59:37,649 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 478ms (threshold=20ms); ack: seqno: 34486 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1432485, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:37,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 34589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 668171, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:38,233 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 446ms (threshold=20ms); ack: seqno: 34653 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1593646, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:38,816 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 477ms (threshold=20ms); ack: seqno: 34819 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1197060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.8571428571428571, tune compaction throughput to 18.57 MB/sec 2015-10-16 12:59:39,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 406ms (threshold=20ms); ack: seqno: 34986 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1331482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:39,402 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 34990 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1099552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:39,567 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 35154 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 727783, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:39,892 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 326ms (threshold=20ms); ack: seqno: 35155 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1394604, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:39,951 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 35162 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1220162, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:40,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630276_1099518554906] hdfs.DFSClient: Slow ReadProcessor read fields took 381ms (threshold=20ms); ack: seqno: 35322 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1444361, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:40,968 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 411ms (threshold=20ms); ack: seqno: 35495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1433399, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:41,506 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 435ms (threshold=20ms); ack: seqno: 35661 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1374092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:42,044 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 434ms (threshold=20ms); ack: seqno: 35828 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1596430, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:42,165 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 35961 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 663667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:42,582 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 396ms (threshold=20ms); ack: seqno: 35995 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1239358, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:43,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 432ms (threshold=20ms); ack: seqno: 36162 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1615958, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:43,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 422ms (threshold=20ms); ack: seqno: 36329 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1445386, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:44,196 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 429ms (threshold=20ms); ack: seqno: 36496 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1284264, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:44,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 428ms (threshold=20ms); ack: seqno: 36663 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1228061, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:44,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 36814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1009334, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:45,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 349ms (threshold=20ms); ack: seqno: 36829 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1419024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:45,810 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 428ms (threshold=20ms); ack: seqno: 36996 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1592359, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:46,338 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 8840ms (threshold=20ms); ack: seqno: 35133 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 938470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:46,349 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 418ms (threshold=20ms); ack: seqno: 37163 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1212487, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:46,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 37258 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 773766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:46,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 35427 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 485329, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:46,454 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:59:46,454 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:59:46,454 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:59:46,454 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:59:46,454 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:59:46,488 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 37265 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 713068, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:46,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 351ms (threshold=20ms); ack: seqno: 37330 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 987031, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:47,071 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630271_1099518554901] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 38337 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 12:59:47,243 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:59:47,292 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025541065 with entries=78072, filesize=122.38 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025587243 2015-10-16 12:59:47,294 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025525687 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025525687 2015-10-16 12:59:47,295 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630281_1099518554911] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1118109, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:47,300 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025533316 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025533316 2015-10-16 12:59:47,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630281_1099518554911] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 441 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 405342, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:47,417 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:59:47,417 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:59:47,417 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:59:47,417 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:59:47,417 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 12:59:47,424 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630280_1099518554910] hdfs.DFSClient: Slow ReadProcessor read fields took 413ms (threshold=20ms); ack: seqno: 37497 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1257642, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 12:59:47,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 400ms (threshold=20ms); ack: seqno: 37670 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1413234, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:48,157 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630281_1099518554911] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 4467 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 421884, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:48,157 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:59:48,157 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:59:48,158 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:59:48,158 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 12:59:48,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 409ms (threshold=20ms); ack: seqno: 37836 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1335879, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:48,998 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630281_1099518554911] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 9260 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 496069, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:48,998 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:48,998 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:48,998 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:48,998 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:48,998 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:49,038 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 401ms (threshold=20ms); ack: seqno: 38005 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1325229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:49,576 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 402ms (threshold=20ms); ack: seqno: 38172 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1313283, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:49,625 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 38176 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1339215, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:49,761 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 38301 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762594, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:49,761 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630281_1099518554911] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 13614 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 458719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:49,761 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:49,761 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:49,761 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:49,761 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:49,761 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:50,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 324ms (threshold=20ms); ack: seqno: 38339 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1254973, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:50,557 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:50,557 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630281_1099518554911] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 18329 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 432052, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:50,558 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:50,558 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:50,558 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:50,558 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:50,652 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 405ms (threshold=20ms); ack: seqno: 38505 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1369516, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:50,678 DEBUG [PriorityRpcServer.handler=17,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:59:50,679 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134278176 > lower bound=16777216 2015-10-16 12:59:50,679 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 12:59:50,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 38510 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1448218, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:51,219 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630281_1099518554911] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 21946 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 415158, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:51,219 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630283_1099518554913] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 605 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 732136, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:51,219 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:51,219 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:51,219 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:51,219 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:51,219 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:51,221 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 256ms (threshold=20ms); ack: seqno: 38672 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1373568, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:51,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8337139, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/bdc75ff6fd244673bc4a1407997c883b 2015-10-16 12:59:51,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/bdc75ff6fd244673bc4a1407997c883b as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bdc75ff6fd244673bc4a1407997c883b 2015-10-16 12:59:51,628 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bdc75ff6fd244673bc4a1407997c883b, entries=466300, sequenceid=8337139, filesize=70.8 M 2015-10-16 12:59:51,629 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134292160, currentsize=28.62 MB/30011600 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 950ms, sequenceid=8337139, compaction requested=true 2015-10-16 12:59:51,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 12:59:51,629 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 5 compacting, 4 eligible, 10 blocking 2015-10-16 12:59:51,631 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 891184735 starting at candidate #0 after considering 3 permutations with 1 in ratio 2015-10-16 12:59:51,631 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 12:59:51,631 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 12:59:51,631 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=849.9 M 2015-10-16 12:59:51,631 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/48c4646896a846498a1f8fdbb315ae3c, keycount=2797530, bloomtype=ROW, size=425.0 M, encoding=NONE, seqNum=8057430 2015-10-16 12:59:51,631 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/65d8ca96dfa84218826aba119767ffd2, keycount=466300, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=8104064 2015-10-16 12:59:51,631 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/31dd1a1f411b41608038c59a0ec77c03, keycount=1864360, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=8290505 2015-10-16 12:59:51,631 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bdc75ff6fd244673bc4a1407997c883b, keycount=466300, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=8337139 2015-10-16 12:59:51,651 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 12:59:51,749 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630281_1099518554911] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 24314 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 563568, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:51,760 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 381ms (threshold=20ms); ack: seqno: 38839 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1235986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:51,815 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 38846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1493933, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:51,881 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:59:51,881 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:59:51,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 38869 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 785110, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:51,881 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 12:59:51,882 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630281_1099518554911] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 24896 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 574488, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:52,711 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630281_1099518554911] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 29823 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 400007, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:52,711 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:52,711 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:52,711 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:52,711 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:52,711 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:52,737 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630284_1099518554914] hdfs.DFSClient: Slow ReadProcessor read fields took 966ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1123452, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:59:52,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 849ms (threshold=20ms); ack: seqno: 39006 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1094961, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:53,033 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 197ms (threshold=20ms); ack: seqno: 39007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2091161, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:53,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630281_1099518554911] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 34015 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 436223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:53,430 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:53,430 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:53,430 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:53,430 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:53,431 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:53,813 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630284_1099518554914] hdfs.DFSClient: Slow ReadProcessor read fields took 954ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1160956, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:59:53,911 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 809ms (threshold=20ms); ack: seqno: 39173 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1169109, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:54,145 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630281_1099518554911] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 38149 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 388951, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:54,145 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:54,145 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:54,145 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:54,145 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:54,145 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:54,270 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 12:59:54,309 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025587243 with entries=77934, filesize=122.17 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025594270 2015-10-16 12:59:54,311 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025541065 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025541065 2015-10-16 12:59:54,864 DEBUG [PriorityRpcServer.handler=4,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:59:54,864 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134266896 > lower bound=16777216 2015-10-16 12:59:54,865 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 12:59:54,889 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630284_1099518554914] hdfs.DFSClient: Slow ReadProcessor read fields took 952ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1213257, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:59:54,895 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630285_1099518554915] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 3395 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 789060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:55,026 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630284_1099518554914] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 604 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 745717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:59:55,026 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630285_1099518554915] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 3909 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 501540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:55,026 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:55,026 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:55,026 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 12:59:55,026 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:55,026 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:55,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 979ms (threshold=20ms); ack: seqno: 39340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1284612, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:55,131 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630286_1099518554916] hdfs.DFSClient: Slow ReadProcessor read fields took 201ms (threshold=20ms); ack: seqno: 16 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 204060028, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:59:55,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630285_1099518554915] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 7064 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 405766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:55,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630286_1099518554916] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 841 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 961981, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:59:55,639 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:59:55,639 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:59:55,639 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:59:55,639 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:59:55,639 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 12:59:55,858 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8383766, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c0eaef4cae524cc1975ae7dcb5172313 2015-10-16 12:59:55,863 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c0eaef4cae524cc1975ae7dcb5172313 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c0eaef4cae524cc1975ae7dcb5172313 2015-10-16 12:59:55,870 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c0eaef4cae524cc1975ae7dcb5172313, entries=466230, sequenceid=8383766, filesize=70.8 M 2015-10-16 12:59:55,872 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134269360, currentsize=26.72 MB/28021920 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1007ms, sequenceid=8383766, compaction requested=false 2015-10-16 12:59:55,965 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630284_1099518554914] hdfs.DFSClient: Slow ReadProcessor read fields took 872ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1173703, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:59:56,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630282_1099518554912] hdfs.DFSClient: Slow ReadProcessor read fields took 877ms (threshold=20ms); ack: seqno: 39507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1190351, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 12:59:56,241 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630285_1099518554915] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 9664 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 705914, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:56,443 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:59:56,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630285_1099518554915] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 10607 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 468760, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:56,443 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:59:56,443 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:59:56,443 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:59:56,443 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 12:59:57,041 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630284_1099518554914] hdfs.DFSClient: Slow ReadProcessor read fields took 928ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1473387, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:59:57,140 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 938ms (threshold=20ms); ack: seqno: 39678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1344957, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:57,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 39767 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887154, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:57,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630285_1099518554915] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 15102 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 479039, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:57,248 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:57,248 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:59:57,248 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:57,248 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:57,248 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 12:59:58,117 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630284_1099518554914] hdfs.DFSClient: Slow ReadProcessor read fields took 931ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1267716, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:59:58,168 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630285_1099518554915] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 20342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 589560, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:58,168 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630284_1099518554914] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 1012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941752, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:59:58,168 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:58,168 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 12:59:58,168 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:58,168 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:58,168 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:58,214 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 905ms (threshold=20ms); ack: seqno: 39844 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1156378, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:58,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630285_1099518554915] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 24518 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 520612, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:58,912 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:58,912 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:58,912 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:58,912 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:58,912 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 12:59:59,194 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630284_1099518554914] hdfs.DFSClient: Slow ReadProcessor read fields took 905ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1352798, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 12:59:59,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 908ms (threshold=20ms); ack: seqno: 40013 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1258110, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 12:59:59,350 DEBUG [PriorityRpcServer.handler=15,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 12:59:59,350 WARN [MemStoreFlusher.0] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 12:59:59,350 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 12:59:59,718 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630285_1099518554915] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 28679 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 445028, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 12:59:59,718 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:59,718 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 12:59:59,718 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:00:00,271 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630284_1099518554914] hdfs.DFSClient: Slow ReadProcessor read fields took 956ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1292862, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:00:00,367 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 946ms (threshold=20ms); ack: seqno: 40180 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1108880, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:00,514 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 40332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873567, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:00,514 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630285_1099518554915] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 32935 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 562820, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:00,514 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:00:00,514 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:00:00,514 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:00:00,514 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:00:00,514 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:00:00,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630285_1099518554915] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 33217 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812881, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:01,346 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630284_1099518554914] hdfs.DFSClient: Slow ReadProcessor read fields took 950ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1521523, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:00:01,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 919ms (threshold=20ms); ack: seqno: 40346 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1171428, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:01,518 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 40369 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 818954, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:01,518 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630285_1099518554915] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 37646 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 531130, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:01,518 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:00:01,518 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:00:01,518 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:00:01,788 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:00:01,833 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025594270 with entries=77856, filesize=122.05 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025601788 2015-10-16 13:00:01,835 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025587243 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025587243 2015-10-16 13:00:01,836 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630288_1099518554918] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1144977, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:00:02,496 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630288_1099518554918] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 3242 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 538667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:00:02,496 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630284_1099518554914] hdfs.DFSClient: Slow ReadProcessor read fields took 1018ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 38643661, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:00:02,496 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:02,496 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:00:02,497 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:00:02,496 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:02,496 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:00:02,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 869ms (threshold=20ms); ack: seqno: 40513 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1171668, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:02,545 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 40519 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1184215, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:03,388 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630288_1099518554918] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 7664 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 653471, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:00:03,389 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:03,389 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:03,389 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:03,389 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:03,389 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:03,499 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630284_1099518554914] hdfs.DFSClient: Slow ReadProcessor read fields took 905ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1181750, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:00:03,595 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 906ms (threshold=20ms); ack: seqno: 40682 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023517, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:04,253 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630288_1099518554918] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 11929 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 572646, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:00:04,253 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:00:04,253 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:04,253 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:00:04,253 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:00:04,253 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:00:04,575 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630284_1099518554914] hdfs.DFSClient: Slow ReadProcessor read fields took 938ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1200969, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:00:04,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 917ms (threshold=20ms); ack: seqno: 40849 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1172497, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:05,117 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630288_1099518554918] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 16133 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 419858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:00:05,117 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:00:05,117 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:00:05,117 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:00:05,117 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:00:05,117 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:00:05,311 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630288_1099518554918] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 16534 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 843999, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:00:05,650 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630289_1099518554919] hdfs.DFSClient: Slow ReadProcessor read fields took 931ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1227295, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:05,747 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 944ms (threshold=20ms); ack: seqno: 41016 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1295434, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:05,790 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 41021 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1009173, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:06,087 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630288_1099518554918] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 20342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 629357, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:00:06,087 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:00:06,087 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:00:06,087 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:00:06,087 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:00:06,087 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:00:06,621 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630288_1099518554918] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 22934 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:00:06,726 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630289_1099518554919] hdfs.DFSClient: Slow ReadProcessor read fields took 949ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1159243, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:06,822 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 911ms (threshold=20ms); ack: seqno: 41183 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010103, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:06,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 41187 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1283857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:07,013 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 41329 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942273, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:07,013 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630288_1099518554918] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 24682 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 559128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:00:07,013 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:07,013 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:07,013 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:07,013 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:07,013 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:07,802 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630289_1099518554919] hdfs.DFSClient: Slow ReadProcessor read fields took 947ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1071269, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:07,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 870ms (threshold=20ms); ack: seqno: 41350 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1321365, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:07,974 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630288_1099518554918] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 29458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838775, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:00:07,974 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630289_1099518554919] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 2656 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919601, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:07,974 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 41376 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 828030, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:07,974 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:00:07,974 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:00:08,878 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630289_1099518554919] hdfs.DFSClient: Slow ReadProcessor read fields took 888ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1230493, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:08,949 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630288_1099518554918] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 34363 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 663492, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:00:08,949 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630289_1099518554919] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 2695 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 753874, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:08,949 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:00:08,949 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:00:08,949 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:00:08,949 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:00:08,949 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:00:08,975 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630287_1099518554917] hdfs.DFSClient: Slow ReadProcessor read fields took 897ms (threshold=20ms); ack: seqno: 41517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1357789, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:09,809 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630288_1099518554918] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 38185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 814530, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:00:09,960 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630288_1099518554918] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 38740 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 582455, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:00:09,961 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:00:09,961 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:00:09,961 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:00:09,961 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:00:09,961 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:00:09,962 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630289_1099518554919] hdfs.DFSClient: Slow ReadProcessor read fields took 873ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1107201, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:10,052 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 920ms (threshold=20ms); ack: seqno: 41688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1416973, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:10,080 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:00:10,124 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025601788 with entries=77946, filesize=122.19 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025610080 2015-10-16 13:00:10,127 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630291_1099518554921] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1095118, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:10,242 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 187ms (threshold=20ms); ack: seqno: 41694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1711363, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:10,408 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 41836 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 820922, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:10,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630291_1099518554921] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 3397 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 514524, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:10,765 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:00:10,765 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:00:10,765 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:00:10,765 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:00:10,765 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:00:11,038 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630289_1099518554919] hdfs.DFSClient: Slow ReadProcessor read fields took 954ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1337600, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:11,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 706ms (threshold=20ms); ack: seqno: 41855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988387, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:11,562 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630291_1099518554921] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 7598 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 542356, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:11,562 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:00:11,562 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:00:11,562 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:00:11,562 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:00:12,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630289_1099518554919] hdfs.DFSClient: Slow ReadProcessor read fields took 959ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1143284, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:12,204 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 941ms (threshold=20ms); ack: seqno: 42021 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1159501, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:12,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 42166 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857555, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:12,358 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:00:12,358 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:00:12,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630291_1099518554921] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 11906 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 532736, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:12,358 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:00:12,358 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:00:13,190 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630289_1099518554919] hdfs.DFSClient: Slow ReadProcessor read fields took 959ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1234915, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:13,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630291_1099518554921] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 16999 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 706568, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:13,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 930ms (threshold=20ms); ack: seqno: 42190 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1237051, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:13,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630289_1099518554919] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 3449 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1084539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:14,267 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630289_1099518554919] hdfs.DFSClient: Slow ReadProcessor read fields took 908ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1546318, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:14,381 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 952ms (threshold=20ms); ack: seqno: 42356 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1289036, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:15,343 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630289_1099518554919] hdfs.DFSClient: Slow ReadProcessor read fields took 971ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1549046, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:15,458 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 967ms (threshold=20ms); ack: seqno: 42523 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1251217, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:15,559 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 42616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 778477, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:16,419 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630289_1099518554919] hdfs.DFSClient: Slow ReadProcessor read fields took 975ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1779528, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:16,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 916ms (threshold=20ms); ack: seqno: 42690 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1392853, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:17,495 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630289_1099518554919] hdfs.DFSClient: Slow ReadProcessor read fields took 976ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1531031, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:17,608 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#36 3313239/5594490 (59.22%), rate=12042.96 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=18.57 MB/sec, activeCompactions=2] 2015-10-16 13:00:17,612 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 954ms (threshold=20ms); ack: seqno: 42857 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2250043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:17,718 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#36 sleep 966 ms because current throughput is 90.91 MB/sec, max allowed is 9.29 MB/sec, already slept 110 time(s) and total slept time is 78523 ms till now. 2015-10-16 13:00:18,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630292_1099518554922] hdfs.DFSClient: Slow ReadProcessor read fields took 958ms (threshold=20ms); ack: seqno: 4181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1673901, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:18,687 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 968ms (threshold=20ms); ack: seqno: 43024 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1297266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:19,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630292_1099518554922] hdfs.DFSClient: Slow ReadProcessor read fields took 965ms (threshold=20ms); ack: seqno: 4348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1446704, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:19,762 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 969ms (threshold=20ms); ack: seqno: 43191 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1282240, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:19,861 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 43288 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751275, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:20,723 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630292_1099518554922] hdfs.DFSClient: Slow ReadProcessor read fields took 966ms (threshold=20ms); ack: seqno: 4517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1446207, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:20,838 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 931ms (threshold=20ms); ack: seqno: 43360 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1496343, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:21,799 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630292_1099518554922] hdfs.DFSClient: Slow ReadProcessor read fields took 974ms (threshold=20ms); ack: seqno: 4684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1567588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:21,915 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 963ms (threshold=20ms); ack: seqno: 43527 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1435507, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:22,991 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630290_1099518554920] hdfs.DFSClient: Slow ReadProcessor read fields took 967ms (threshold=20ms); ack: seqno: 43694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1553237, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:23,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630292_1099518554922] hdfs.DFSClient: Slow ReadProcessor read fields took 1295ms (threshold=20ms); ack: seqno: 4851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 320740343, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:23,952 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630292_1099518554922] hdfs.DFSClient: Slow ReadProcessor read fields took 703ms (threshold=20ms); ack: seqno: 5018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1593371, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:24,067 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630293_1099518554923] hdfs.DFSClient: Slow ReadProcessor read fields took 961ms (threshold=20ms); ack: seqno: 43865 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1554905, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:24,174 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630293_1099518554923] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 43979 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741468, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:25,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630292_1099518554922] hdfs.DFSClient: Slow ReadProcessor read fields took 975ms (threshold=20ms); ack: seqno: 5185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1560977, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:25,142 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630293_1099518554923] hdfs.DFSClient: Slow ReadProcessor read fields took 935ms (threshold=20ms); ack: seqno: 44032 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1248400, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:26,105 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630292_1099518554922] hdfs.DFSClient: Slow ReadProcessor read fields took 974ms (threshold=20ms); ack: seqno: 5352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1435662, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:26,218 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630293_1099518554923] hdfs.DFSClient: Slow ReadProcessor read fields took 969ms (threshold=20ms); ack: seqno: 44197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1350159, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:27,181 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630292_1099518554922] hdfs.DFSClient: Slow ReadProcessor read fields took 977ms (threshold=20ms); ack: seqno: 5521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1552631, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:27,294 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630293_1099518554923] hdfs.DFSClient: Slow ReadProcessor read fields took 964ms (threshold=20ms); ack: seqno: 44366 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1203448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:28,266 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630292_1099518554922] hdfs.DFSClient: Slow ReadProcessor read fields took 985ms (threshold=20ms); ack: seqno: 5688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 9582677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:28,370 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630293_1099518554923] hdfs.DFSClient: Slow ReadProcessor read fields took 969ms (threshold=20ms); ack: seqno: 44533 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1334097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:28,494 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630293_1099518554923] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 44674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 675961, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:29,334 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630292_1099518554922] hdfs.DFSClient: Slow ReadProcessor read fields took 974ms (threshold=20ms); ack: seqno: 5855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1540208, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:29,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630293_1099518554923] hdfs.DFSClient: Slow ReadProcessor read fields took 935ms (threshold=20ms); ack: seqno: 44700 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1322614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:30,410 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630292_1099518554922] hdfs.DFSClient: Slow ReadProcessor read fields took 956ms (threshold=20ms); ack: seqno: 6022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1500060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:30,522 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630293_1099518554923] hdfs.DFSClient: Slow ReadProcessor read fields took 968ms (threshold=20ms); ack: seqno: 44867 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1109848, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:31,486 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630292_1099518554922] hdfs.DFSClient: Slow ReadProcessor read fields took 961ms (threshold=20ms); ack: seqno: 6189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1443307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:00:31,599 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630293_1099518554923] hdfs.DFSClient: Slow ReadProcessor read fields took 970ms (threshold=20ms); ack: seqno: 45034 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1385761, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:31,600 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630294_1099518554924] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 6252 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1545024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:00:32,561 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630294_1099518554924] hdfs.DFSClient: Slow ReadProcessor read fields took 930ms (threshold=20ms); ack: seqno: 6358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1431408, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:00:32,676 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630293_1099518554923] hdfs.DFSClient: Slow ReadProcessor read fields took 960ms (threshold=20ms); ack: seqno: 45201 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1167723, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:32,791 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630293_1099518554923] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 45330 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 730130, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:33,637 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630294_1099518554924] hdfs.DFSClient: Slow ReadProcessor read fields took 962ms (threshold=20ms); ack: seqno: 6525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1453753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:00:33,752 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630293_1099518554923] hdfs.DFSClient: Slow ReadProcessor read fields took 935ms (threshold=20ms); ack: seqno: 45368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1241771, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:34,714 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630294_1099518554924] hdfs.DFSClient: Slow ReadProcessor read fields took 973ms (threshold=20ms); ack: seqno: 6694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1701566, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:00:34,828 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630293_1099518554923] hdfs.DFSClient: Slow ReadProcessor read fields took 965ms (threshold=20ms); ack: seqno: 45535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1552104, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:35,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630294_1099518554924] hdfs.DFSClient: Slow ReadProcessor read fields took 974ms (threshold=20ms); ack: seqno: 6861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1583167, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:00:35,903 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630293_1099518554923] hdfs.DFSClient: Slow ReadProcessor read fields took 971ms (threshold=20ms); ack: seqno: 45702 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1171327, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:00:36,866 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630294_1099518554924] hdfs.DFSClient: Slow ReadProcessor read fields took 976ms (threshold=20ms); ack: seqno: 7028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1465000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:00:36,979 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630295_1099518554925] hdfs.DFSClient: Slow ReadProcessor read fields took 970ms (threshold=20ms); ack: seqno: 45873 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1281979, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:00:37,116 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630295_1099518554925] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 46028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 643294, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:00:37,955 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630294_1099518554924] hdfs.DFSClient: Slow ReadProcessor read fields took 989ms (threshold=20ms); ack: seqno: 7195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1714154, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:00:38,056 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630295_1099518554925] hdfs.DFSClient: Slow ReadProcessor read fields took 929ms (threshold=20ms); ack: seqno: 46042 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1380018, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:00:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 13:00:39,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630294_1099518554924] hdfs.DFSClient: Slow ReadProcessor read fields took 952ms (threshold=20ms); ack: seqno: 7362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1618325, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:00:39,130 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630295_1099518554925] hdfs.DFSClient: Slow ReadProcessor read fields took 965ms (threshold=20ms); ack: seqno: 46207 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1213514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:00:39,450 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630294_1099518554924] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 7980 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 686547, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:00:39,465 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630295_1099518554925] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 46610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 889067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:00:39,922 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630296_1099518554926] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 8684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811989, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:00:39,922 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630295_1099518554925] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 47258 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769543, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:00:40,404 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630296_1099518554926] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 9359 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741245, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:00:40,404 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630295_1099518554925] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 47928 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 664822, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:00:40,877 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630296_1099518554926] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 10056 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736712, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:00:40,877 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630297_1099518554927] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 48601 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 776872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:00:41,381 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630297_1099518554927] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 49289 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885020, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:00:41,381 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630298_1099518554928] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 10718 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013299, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:00:41,467 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#36 average throughput is 20.32 MB/sec, slept 130 time(s) and total slept time is 97658 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 13:00:41,489 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1f08a83c576d4583bc9945047a56a4d7 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1f08a83c576d4583bc9945047a56a4d7 2015-10-16 13:00:41,504 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630291_1099518554921] hdfs.DFSClient: Slow ReadProcessor read fields took 27846ms (threshold=20ms); ack: seqno: 18984 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 542036, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:41,504 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:00:41,508 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:00:41,513 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/97dc7feeac8749ef8d85f3ae08fe46a4, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/97dc7feeac8749ef8d85f3ae08fe46a4 2015-10-16 13:00:41,522 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/71c97a8cbb7a48cfb9abc6df2f566e4f, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/71c97a8cbb7a48cfb9abc6df2f566e4f 2015-10-16 13:00:41,530 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/008ad9db86cb441b8299e952f6e19a04, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/008ad9db86cb441b8299e952f6e19a04 2015-10-16 13:00:41,538 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cbce6832938342ad97e6c43061301946, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/cbce6832938342ad97e6c43061301946 2015-10-16 13:00:41,547 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/94019a24eb154262b7eab1645e409d30, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/94019a24eb154262b7eab1645e409d30 2015-10-16 13:00:41,547 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 1f08a83c576d4583bc9945047a56a4d7(size=3.0 G), total size for store is 12.4 G. This selection was in queue for 0sec, and took 2mins, 25sec to execute. 2015-10-16 13:00:41,547 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=5, fileSize=3.0 G, priority=4, time=36903405003365549; duration=2mins, 25sec 2015-10-16 13:00:41,547 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:00:41,547 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 4 compacting, 1 eligible, 10 blocking 2015-10-16 13:00:41,548 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:00:41,548 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:00:41,548 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:00:41,664 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536917216 > lower bound=16777216 2015-10-16 13:00:41,664 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=512.04 MB 2015-10-16 13:00:41,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630291_1099518554921] hdfs.DFSClient: Slow ReadProcessor read fields took 262ms (threshold=20ms); ack: seqno: 18985 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 484467, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:42,055 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630299_1099518554929] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 748311, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:00:42,055 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630298_1099518554928] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 11719 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 696555, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:00:42,642 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630298_1099518554928] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 12502 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 42555772, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:00:42,696 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630299_1099518554929] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1489 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797423, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:00:42,696 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630300_1099518554930] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 12547 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 763507, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:00:43,357 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630300_1099518554930] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 13456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 659316, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:00:43,357 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630301_1099518554931] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 2570 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 734957, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:00:43,602 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#39 average throughput is 15.91 MB/sec, slept 44 time(s) and total slept time is 41512 ms. 0 active compactions remaining, total limit is unlimited 2015-10-16 13:00:43,620 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/feb37f4f73a84697a2c896c9c77546e0 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/feb37f4f73a84697a2c896c9c77546e0 2015-10-16 13:00:43,636 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630291_1099518554921] hdfs.DFSClient: Slow ReadProcessor read fields took 1869ms (threshold=20ms); ack: seqno: 18986 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 587954, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:43,636 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:00:43,637 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:00:43,647 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/48c4646896a846498a1f8fdbb315ae3c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/48c4646896a846498a1f8fdbb315ae3c 2015-10-16 13:00:43,655 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/65d8ca96dfa84218826aba119767ffd2, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/65d8ca96dfa84218826aba119767ffd2 2015-10-16 13:00:43,663 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/31dd1a1f411b41608038c59a0ec77c03, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/31dd1a1f411b41608038c59a0ec77c03 2015-10-16 13:00:43,672 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bdc75ff6fd244673bc4a1407997c883b, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bdc75ff6fd244673bc4a1407997c883b 2015-10-16 13:00:43,672 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into feb37f4f73a84697a2c896c9c77546e0(size=849.8 M), total size for store is 12.4 G. This selection was in queue for 0sec, and took 52sec to execute. 2015-10-16 13:00:43,672 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=849.9 M, priority=0, time=36903500485110799; duration=52sec 2015-10-16 13:00:43,672 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:00:44,638 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8570206, memsize=512.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8a5a14b2487443aa95ca216d269ae027 2015-10-16 13:00:44,645 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8a5a14b2487443aa95ca216d269ae027 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8a5a14b2487443aa95ca216d269ae027 2015-10-16 13:00:44,650 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8a5a14b2487443aa95ca216d269ae027, entries=1864350, sequenceid=8570206, filesize=283.2 M 2015-10-16 13:00:44,652 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630291_1099518554921] hdfs.DFSClient: Slow ReadProcessor read fields took 1016ms (threshold=20ms); ack: seqno: 18987 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 624899, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:44,652 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~512.04 MB/536916800, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2988ms, sequenceid=8570206, compaction requested=true 2015-10-16 13:00:44,652 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:00:44,652 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 5 store files, 0 compacting, 5 eligible, 10 blocking 2015-10-16 13:00:44,655 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 6 permutations with 0 in ratio 2015-10-16 13:00:44,655 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:00:44,655 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:00:51,867 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630291_1099518554921] hdfs.DFSClient: Slow ReadProcessor read fields took 7216ms (threshold=20ms); ack: seqno: 18988 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043084, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:52,083 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630291_1099518554921] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 19436 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834547, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:53,034 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630291_1099518554921] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 24672 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 477065, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:53,034 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:00:53,034 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:00:53,034 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:00:53,034 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:00:53,034 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:00:53,990 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 58ms (threshold=20ms) 2015-10-16 13:00:53,990 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630291_1099518554921] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 30229 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 419739, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:53,990 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 58ms (threshold=20ms) 2015-10-16 13:00:53,990 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 58ms (threshold=20ms) 2015-10-16 13:00:53,990 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 58ms (threshold=20ms) 2015-10-16 13:00:55,001 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630291_1099518554921] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 35741 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 566280, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:55,001 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:00:55,001 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:00:55,001 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:00:55,001 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:00:55,001 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:00:55,720 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630291_1099518554921] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 39081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908019, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:55,777 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:00:55,830 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025610080 with entries=77873, filesize=122.07 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025655777 2015-10-16 13:00:55,831 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025594270 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025594270 2015-10-16 13:00:55,833 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630303_1099518554933] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022964, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:00:55,838 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025601788 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025601788 2015-10-16 13:00:56,206 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630303_1099518554933] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 1799 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 487421, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:00:56,206 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:00:56,206 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:00:56,206 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:00:56,206 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:00:56,206 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:00:56,440 DEBUG [PriorityRpcServer.handler=0,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:00:56,440 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134279056 > lower bound=16777216 2015-10-16 13:00:56,440 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 13:00:56,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630303_1099518554933] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 3085 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 662565, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:00:57,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630304_1099518554934] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 775 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1201085, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:00:57,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630303_1099518554933] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 5853 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 594875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:00:57,011 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:57,011 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:57,011 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:57,011 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:57,011 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:00:57,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8616838, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/512d8ef54895466a8e824a84348006ce 2015-10-16 13:00:57,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/512d8ef54895466a8e824a84348006ce as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/512d8ef54895466a8e824a84348006ce 2015-10-16 13:00:57,290 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/512d8ef54895466a8e824a84348006ce, entries=466270, sequenceid=8616838, filesize=70.8 M 2015-10-16 13:00:57,291 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134281520, currentsize=23.55 MB/24689440 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 851ms, sequenceid=8616838, compaction requested=true 2015-10-16 13:00:57,292 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:00:57,292 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 6 store files, 0 compacting, 6 eligible, 10 blocking 2015-10-16 13:00:57,294 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 10 permutations with 0 in ratio 2015-10-16 13:00:57,294 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:00:57,294 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:00:57,868 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630303_1099518554933] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 10619 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 484927, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:00:57,869 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:00:57,869 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:00:57,869 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:00:57,869 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:00:57,869 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:00:58,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630303_1099518554933] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 16195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 592097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:00:58,879 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:00:58,879 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:00:58,879 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:00:58,880 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:00:59,385 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630303_1099518554933] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 18499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839129, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:00:59,989 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630303_1099518554933] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 21742 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 447084, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:00:59,989 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:00:59,989 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:00:59,989 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:00:59,989 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:01:00,879 DEBUG [PriorityRpcServer.handler=5,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:01:00,879 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134277472 > lower bound=16777216 2015-10-16 13:01:00,879 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:01:00,907 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630303_1099518554933] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 26612 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 723450, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:01,043 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630305_1099518554935] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 73 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3761631, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:01,043 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630303_1099518554933] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 27036 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672028, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:01,043 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:01:01,043 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:01:01,043 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:01:01,043 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:01:01,043 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:01:01,729 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8663471, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/788fc083c30f4be8b6651f6c22f75331 2015-10-16 13:01:01,734 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/788fc083c30f4be8b6651f6c22f75331 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/788fc083c30f4be8b6651f6c22f75331 2015-10-16 13:01:01,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630303_1099518554933] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 30608 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 441284, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:01,783 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:01:01,783 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:01:01,783 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:01:01,783 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:01:01,783 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:01:01,785 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/788fc083c30f4be8b6651f6c22f75331, entries=466290, sequenceid=8663471, filesize=70.8 M 2015-10-16 13:01:01,786 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134288000, currentsize=22.35 MB/23439520 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 907ms, sequenceid=8663471, compaction requested=true 2015-10-16 13:01:01,787 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:01:01,787 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 0 compacting, 7 eligible, 10 blocking 2015-10-16 13:01:01,790 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 15 permutations with 0 in ratio 2015-10-16 13:01:01,790 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:01:01,790 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:01:02,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630303_1099518554933] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 36044 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 473916, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:02,789 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:02,789 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:02,790 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:02,790 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:02,790 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:03,303 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:01:03,337 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025655777 with entries=77908, filesize=122.13 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025663303 2015-10-16 13:01:03,341 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025610080 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025610080 2015-10-16 13:01:03,776 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 1952 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 775680, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:03,893 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 2381 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 337305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:03,893 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:01:03,893 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:01:03,893 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:01:03,893 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:01:04,866 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 7948 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 775831, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:04,866 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:01:04,866 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:01:05,356 DEBUG [PriorityRpcServer.handler=12,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:01:05,356 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134258496 > lower bound=16777216 2015-10-16 13:01:05,356 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 13:01:05,734 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630307_1099518554937] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 526 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 893622, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:01:05,734 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 12479 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 764156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:05,734 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:05,734 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:05,734 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:05,734 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:05,734 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:05,838 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 12717 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 747835, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:06,179 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8710097, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/57e69239fb014fba8bce3940c3e965f1 2015-10-16 13:01:06,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/57e69239fb014fba8bce3940c3e965f1 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/57e69239fb014fba8bce3940c3e965f1 2015-10-16 13:01:06,198 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/57e69239fb014fba8bce3940c3e965f1, entries=466220, sequenceid=8710097, filesize=70.8 M 2015-10-16 13:01:06,200 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134266720, currentsize=21.41 MB/22449200 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 844ms, sequenceid=8710097, compaction requested=true 2015-10-16 13:01:06,200 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:01:06,200 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 0 compacting, 8 eligible, 10 blocking 2015-10-16 13:01:06,204 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 5 files of size 594077578 starting at candidate #3 after considering 21 permutations with 2 in ratio 2015-10-16 13:01:06,204 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:01:06,204 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:01:06,204 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=566.6 M 2015-10-16 13:01:06,204 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c0eaef4cae524cc1975ae7dcb5172313, keycount=466230, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=8383766 2015-10-16 13:01:06,204 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8a5a14b2487443aa95ca216d269ae027, keycount=1864350, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=8570206 2015-10-16 13:01:06,205 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/512d8ef54895466a8e824a84348006ce, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=8616838 2015-10-16 13:01:06,205 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/788fc083c30f4be8b6651f6c22f75331, keycount=466290, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=8663471 2015-10-16 13:01:06,205 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/57e69239fb014fba8bce3940c3e965f1, keycount=466220, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=8710097 2015-10-16 13:01:06,231 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:01:06,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630308_1099518554938] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 234 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842121, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:06,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 15762 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 399102, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:06,442 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:06,442 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:06,442 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:06,442 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:06,442 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:06,979 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630308_1099518554938] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:06,979 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 18631 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 472705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:06,979 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:06,979 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:06,979 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:06,979 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:06,979 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:07,517 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630308_1099518554938] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 1564 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 626187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:07,517 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 21413 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 449640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:07,517 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:07,517 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:07,517 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:07,518 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:07,664 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 13:01:07,664 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 13:01:07,664 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 13:01:07,664 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 23ms (threshold=20ms) 2015-10-16 13:01:08,042 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:08,042 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630309_1099518554939] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 2275 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736883, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:01:08,043 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 23989 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 575783, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:08,557 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 26609 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 479116, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:08,557 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630309_1099518554939] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 2985 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 815908, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:01:08,557 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:08,557 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:08,557 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:09,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 29266 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 517880, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:09,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630309_1099518554939] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 3691 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 631530, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:01:09,080 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:09,080 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:09,080 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:09,080 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:09,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 29327 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 818943, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:09,683 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630310_1099518554940] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 4458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 855094, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:01:09,683 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 31712 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 593060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:09,683 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:01:09,683 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:01:09,683 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:01:09,683 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:01:09,683 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:01:10,118 DEBUG [PriorityRpcServer.handler=19,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:01:10,118 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134288208 > lower bound=16777216 2015-10-16 13:01:10,118 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.03 MB 2015-10-16 13:01:10,203 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 34402 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 437384, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:10,203 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630310_1099518554940] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 5136 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:01:10,203 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:10,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 36507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 523652, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:10,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630311_1099518554941] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 675 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 855512, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:10,660 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630310_1099518554940] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 5665 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905314, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:01:10,660 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:10,660 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:10,660 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:10,660 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:10,660 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:10,987 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8756735, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/860e1764c9fc4cfabb66b883cb15b410 2015-10-16 13:01:10,992 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/860e1764c9fc4cfabb66b883cb15b410 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/860e1764c9fc4cfabb66b883cb15b410 2015-10-16 13:01:10,999 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/860e1764c9fc4cfabb66b883cb15b410, entries=466340, sequenceid=8756735, filesize=70.8 M 2015-10-16 13:01:11,001 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134302480, currentsize=21.91 MB/22973360 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 883ms, sequenceid=8756735, compaction requested=true 2015-10-16 13:01:11,001 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:01:11,001 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 5 compacting, 1 eligible, 10 blocking 2015-10-16 13:01:11,002 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:01:11,002 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:01:11,002 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:01:11,131 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630310_1099518554940] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 6242 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 640753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:01:11,131 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630306_1099518554936] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 38783 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 417121, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:01:11,131 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:11,131 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:11,131 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:11,179 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:01:11,212 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025663303 with entries=77841, filesize=122.02 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025671179 2015-10-16 13:01:11,214 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025655777 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025655777 2015-10-16 13:01:11,266 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 13:01:11,691 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630312_1099518554942] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 6996 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808236, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:01:11,691 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:11,691 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:11,691 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:11,691 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:11,691 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:12,264 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 62ms (threshold=20ms) 2015-10-16 13:01:12,264 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630313_1099518554943] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 4818 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 408559, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:12,264 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630312_1099518554942] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 7682 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 817298, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:01:12,264 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 13:01:12,807 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630313_1099518554943] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 7542 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 567103, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:12,807 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:12,807 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:12,807 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:12,807 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:12,807 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:13,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630313_1099518554943] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 10192 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 627784, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:13,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630314_1099518554944] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 9085 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 704239, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:13,355 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:13,355 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:13,355 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:13,355 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:13,355 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:13,445 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#40 average throughput is 76.42 MB/sec, slept 0 time(s) and total slept time is 0 ms. 0 active compactions remaining, total limit is unlimited 2015-10-16 13:01:13,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630314_1099518554944] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 9222 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1165420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:13,577 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hdfs.DFSClient: Slow waitForAckedSeqno took 130ms (threshold=20ms) 2015-10-16 13:01:13,591 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1a8ce525811e466586be9d44ce7aa809 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1a8ce525811e466586be9d44ce7aa809 2015-10-16 13:01:13,600 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:01:13,604 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:01:13,611 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c0eaef4cae524cc1975ae7dcb5172313, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c0eaef4cae524cc1975ae7dcb5172313 2015-10-16 13:01:13,620 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8a5a14b2487443aa95ca216d269ae027, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8a5a14b2487443aa95ca216d269ae027 2015-10-16 13:01:13,628 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/512d8ef54895466a8e824a84348006ce, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/512d8ef54895466a8e824a84348006ce 2015-10-16 13:01:13,636 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/788fc083c30f4be8b6651f6c22f75331, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/788fc083c30f4be8b6651f6c22f75331 2015-10-16 13:01:13,645 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/57e69239fb014fba8bce3940c3e965f1, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/57e69239fb014fba8bce3940c3e965f1 2015-10-16 13:01:13,645 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 1a8ce525811e466586be9d44ce7aa809(size=566.5 M), total size for store is 13.0 G. This selection was in queue for 0sec, and took 7sec to execute. 2015-10-16 13:01:13,645 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=5, fileSize=566.6 M, priority=2, time=36903575058155145; duration=7sec 2015-10-16 13:01:13,645 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:01:14,353 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630313_1099518554943] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 15235 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 501345, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:14,353 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:14,353 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:14,353 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:14,353 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:14,353 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:14,929 DEBUG [PriorityRpcServer.handler=2,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:01:14,929 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134276096 > lower bound=16777216 2015-10-16 13:01:14,929 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 13:01:14,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630313_1099518554943] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 18500 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 669335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:15,283 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630313_1099518554943] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 19887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 563159, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:15,283 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:15,283 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630315_1099518554945] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 433 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 594551, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:01:15,283 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:15,283 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:15,283 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:15,283 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:15,761 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8803365, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/6ddaf1fa9a24403ca59be04a8a37a783 2015-10-16 13:01:15,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/6ddaf1fa9a24403ca59be04a8a37a783 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6ddaf1fa9a24403ca59be04a8a37a783 2015-10-16 13:01:15,773 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6ddaf1fa9a24403ca59be04a8a37a783, entries=466250, sequenceid=8803365, filesize=70.8 M 2015-10-16 13:01:15,774 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134275680, currentsize=22.43 MB/23523440 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 845ms, sequenceid=8803365, compaction requested=true 2015-10-16 13:01:15,774 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:01:15,774 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 6 store files, 0 compacting, 6 eligible, 10 blocking 2015-10-16 13:01:15,776 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 1633629703 starting at candidate #2 after considering 10 permutations with 1 in ratio 2015-10-16 13:01:15,777 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:01:15,777 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:01:15,777 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=1.5 G 2015-10-16 13:01:15,777 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/feb37f4f73a84697a2c896c9c77546e0, keycount=5594490, bloomtype=ROW, size=849.8 M, encoding=NONE, seqNum=8337139 2015-10-16 13:01:15,777 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1a8ce525811e466586be9d44ce7aa809, keycount=3729360, bloomtype=ROW, size=566.5 M, encoding=NONE, seqNum=8710097 2015-10-16 13:01:15,777 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/860e1764c9fc4cfabb66b883cb15b410, keycount=466340, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=8756735 2015-10-16 13:01:15,777 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6ddaf1fa9a24403ca59be04a8a37a783, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=8803365 2015-10-16 13:01:15,796 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:01:15,981 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630313_1099518554943] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 23007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971319, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:16,144 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630313_1099518554943] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 23649 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 622056, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:16,144 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:16,144 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:16,144 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:16,144 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:16,144 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:16,297 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630313_1099518554943] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 24283 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 32329992, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:16,297 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:16,297 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:16,297 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:16,297 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:16,297 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:16,862 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630313_1099518554943] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 27087 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 708125, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:16,862 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630316_1099518554946] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 597 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751604, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:16,862 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:16,862 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:16,862 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:16,862 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:16,862 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:17,405 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:17,405 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:17,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630316_1099518554946] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 1335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 731181, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:17,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630313_1099518554943] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 29658 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 472274, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:17,405 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:17,406 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:17,406 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:17,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630316_1099518554946] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 2038 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692976, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:17,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630313_1099518554943] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 32319 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 663711, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:17,957 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:17,957 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:17,958 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:17,958 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:18,506 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630317_1099518554947] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 2777 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 778311, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:01:18,506 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630313_1099518554943] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 34886 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 513713, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:18,507 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:18,507 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:18,507 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:18,507 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:18,507 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:19,057 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630317_1099518554947] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 3525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 752727, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:01:19,057 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630313_1099518554943] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 37425 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845604, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:19,057 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:01:19,057 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:19,057 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:01:19,057 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:01:19,057 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:01:19,351 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:01:19,395 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025671179 with entries=77886, filesize=122.09 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025679351 2015-10-16 13:01:19,397 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025663303 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025663303 2015-10-16 13:01:19,398 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012318, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:19,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 484309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:19,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630319_1099518554949] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 4227 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2050692, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:19,604 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:19,604 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:19,604 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:19,604 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:19,604 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:19,912 DEBUG [PriorityRpcServer.handler=7,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:01:19,912 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134276976 > lower bound=16777216 2015-10-16 13:01:19,912 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:01:19,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 2880 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 438114, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:20,103 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630320_1099518554950] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 793289, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:20,103 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630319_1099518554949] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 4870 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 803521, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:20,103 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 3569 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 444914, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:20,103 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:20,103 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:20,103 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:20,103 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:20,103 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:20,543 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 5745 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 532949, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:20,543 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630320_1099518554950] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 826170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:20,543 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:20,543 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:20,543 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:20,543 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630319_1099518554949] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 5406 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792963, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:20,543 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:20,543 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:20,794 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8849996, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ae383411b03f466a8a9c579dfd503a31 2015-10-16 13:01:20,799 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ae383411b03f466a8a9c579dfd503a31 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ae383411b03f466a8a9c579dfd503a31 2015-10-16 13:01:20,806 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ae383411b03f466a8a9c579dfd503a31, entries=466270, sequenceid=8849996, filesize=70.8 M 2015-10-16 13:01:20,807 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134282320, currentsize=23.99 MB/25159280 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 895ms, sequenceid=8849996, compaction requested=true 2015-10-16 13:01:20,807 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:01:20,807 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 4 compacting, 1 eligible, 10 blocking 2015-10-16 13:01:20,808 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:01:20,808 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:01:20,808 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:01:21,009 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 8117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 580267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:21,009 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630319_1099518554949] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 6004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 680659, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:01:21,009 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:21,009 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:21,009 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:21,009 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:21,009 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:21,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 10787 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 605753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:21,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630321_1099518554951] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 6713 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758556, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:21,526 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:21,526 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:21,526 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:21,526 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:21,526 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:22,047 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 13475 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 425593, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:22,047 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630321_1099518554951] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 7415 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 760757, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:22,047 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:22,047 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:22,047 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:22,047 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:22,047 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:22,503 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 15393 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 761000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:22,616 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 15835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 513640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:22,616 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630321_1099518554951] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 8193 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 776505, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:22,616 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:22,616 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:22,616 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:22,616 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:22,616 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:23,162 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:23,162 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:23,162 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 18577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 547335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:23,162 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630322_1099518554952] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 8881 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 810142, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:23,162 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:23,163 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:23,163 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:23,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630322_1099518554952] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 9574 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 731861, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:23,713 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 21269 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 717338, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:23,713 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:23,713 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:24,269 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 23970 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 536605, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:24,269 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630322_1099518554952] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 10274 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 749210, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:24,269 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:24,269 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:24,269 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:24,269 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:24,709 DEBUG [PriorityRpcServer.handler=4,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:01:24,709 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134277760 > lower bound=16777216 2015-10-16 13:01:24,709 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:01:24,737 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 26271 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 620416, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:24,814 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630323_1099518554953] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 10987 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 771466, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:01:24,814 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630324_1099518554954] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 50 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 893881, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:24,814 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 26473 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 659721, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:24,814 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:24,814 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:24,815 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:24,815 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:24,815 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:25,193 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630323_1099518554953] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 11436 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 27170204, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:01:25,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630324_1099518554954] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 656 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 749602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:25,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630323_1099518554953] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 11535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:01:25,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 28616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 525721, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:25,272 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:01:25,272 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:01:25,272 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:01:25,272 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:25,273 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:01:25,627 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8896626, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/3448acfb80904a5daab336cb049ae08c 2015-10-16 13:01:25,633 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/3448acfb80904a5daab336cb049ae08c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3448acfb80904a5daab336cb049ae08c 2015-10-16 13:01:25,639 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3448acfb80904a5daab336cb049ae08c, entries=466260, sequenceid=8896626, filesize=70.8 M 2015-10-16 13:01:25,640 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134279360, currentsize=22.98 MB/24096080 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 931ms, sequenceid=8896626, compaction requested=true 2015-10-16 13:01:25,640 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:01:25,641 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 4 compacting, 2 eligible, 10 blocking 2015-10-16 13:01:25,641 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:01:25,641 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:01:25,641 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:01:25,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 30741 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 693940, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:25,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630323_1099518554953] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 12135 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 781423, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:01:25,731 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:01:25,731 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:01:25,731 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:01:25,731 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:01:25,731 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:01:26,253 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630325_1099518554955] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 12826 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782249, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:26,253 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 33463 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 570996, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:26,253 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:26,253 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:26,253 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:26,253 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:26,253 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:26,779 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630325_1099518554955] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 13523 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 659257, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:26,779 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 36166 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 584558, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:26,779 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:26,780 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:26,780 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:26,780 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:27,321 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630325_1099518554955] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 14237 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 871129, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:27,321 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630318_1099518554948] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 38853 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 483827, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:01:27,321 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:27,321 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:27,321 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:01:27,321 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:27,321 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:01:27,340 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:01:27,369 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025679351 with entries=77810, filesize=121.97 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025687340 2015-10-16 13:01:27,371 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025671179 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025671179 2015-10-16 13:01:27,633 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1343 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 614824, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:27,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 2446 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 399485, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:27,864 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:27,864 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:27,864 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:27,864 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:27,864 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:27,865 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630327_1099518554957] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 14937 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965173, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:28,388 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 5210 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 444507, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:28,389 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:28,389 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630327_1099518554957] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 15613 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 872582, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:28,389 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:28,389 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:28,389 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:28,389 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:28,723 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 6398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 650322, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:28,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 7651 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 490369, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:28,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630327_1099518554957] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 16363 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 817842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:28,995 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:28,995 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:28,995 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:28,995 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:28,995 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:29,543 DEBUG [PriorityRpcServer.handler=2,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:01:29,543 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134258224 > lower bound=16777216 2015-10-16 13:01:29,544 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:01:29,573 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 10546 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 556164, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:29,619 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 10603 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 532864, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:29,620 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:29,620 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:29,620 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:29,665 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630328_1099518554958] hdfs.DFSClient: Slow ReadProcessor read fields took 278ms (threshold=20ms); ack: seqno: 16905 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 277965233, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:30,087 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630329_1099518554959] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 637 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813481, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:01:30,087 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630328_1099518554958] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 17482 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 680477, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:30,088 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:30,087 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:30,087 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:30,088 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:30,089 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 12767 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672826, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:30,089 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:30,444 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8943252, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/30a2056e502449f2b17df78d05b011db 2015-10-16 13:01:30,450 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/30a2056e502449f2b17df78d05b011db as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/30a2056e502449f2b17df78d05b011db 2015-10-16 13:01:30,455 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/30a2056e502449f2b17df78d05b011db, entries=466220, sequenceid=8943252, filesize=70.8 M 2015-10-16 13:01:30,457 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134267600, currentsize=22.50 MB/23598000 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 914ms, sequenceid=8943252, compaction requested=true 2015-10-16 13:01:30,458 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:01:30,458 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 4 compacting, 3 eligible, 10 blocking 2015-10-16 13:01:30,459 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222817884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 13:01:30,459 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:01:30,459 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:01:30,459 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 13:01:30,459 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ae383411b03f466a8a9c579dfd503a31, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=8849996 2015-10-16 13:01:30,459 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3448acfb80904a5daab336cb049ae08c, keycount=466260, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=8896626 2015-10-16 13:01:30,459 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/30a2056e502449f2b17df78d05b011db, keycount=466220, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=8943252 2015-10-16 13:01:30,475 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:01:30,549 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630328_1099518554958] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 18042 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 744481, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:30,549 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 14763 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 474912, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:30,549 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630330_1099518554960] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 33 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1754856, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:30,549 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:30,549 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:30,549 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:30,549 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:30,549 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:30,934 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630328_1099518554958] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 18522 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1125799, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:30,934 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630330_1099518554960] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 503 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1046124, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:30,934 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 16456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 445244, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:30,934 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:30,935 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:30,935 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:30,935 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:30,935 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:31,314 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630330_1099518554960] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 959 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950910, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:31,314 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630331_1099518554961] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 18996 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 727403, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:01:31,314 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 18163 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 589314, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:31,314 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:31,314 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:31,314 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:31,314 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:31,693 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:31,693 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630331_1099518554961] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 19464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 786197, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:01:31,693 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 19900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 717553, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:31,693 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630330_1099518554960] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 1410 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920626, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:31,694 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:31,693 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:31,694 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:31,694 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:32,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 21602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 424294, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:32,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630331_1099518554961] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 19940 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 700305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:01:32,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630330_1099518554960] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 1874 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 681211, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:32,078 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:32,078 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:32,078 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:32,078 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:32,078 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:32,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630332_1099518554962] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 2339 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 720763, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:32,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 23292 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 550302, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:32,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630331_1099518554961] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 20406 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769082, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:01:32,455 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:32,455 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:32,455 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:32,455 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:32,456 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:32,834 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630333_1099518554963] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 20849 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2948704, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:01:32,834 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 25032 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 584203, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:32,834 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630332_1099518554962] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 2812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 843645, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:32,834 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:32,834 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:32,834 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:32,834 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:32,834 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:33,213 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 26719 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 522847, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:33,213 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630333_1099518554963] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 21307 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812770, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:01:33,213 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630332_1099518554962] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 3306 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 695866, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:33,213 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:33,213 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:33,213 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:33,213 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:33,213 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:33,320 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#42 average throughput is 72.90 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 13:01:33,697 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630333_1099518554963] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 21902 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813142, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:01:33,697 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 29202 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 578757, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:33,697 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:33,697 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:33,697 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:33,697 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:33,697 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:33,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630332_1099518554962] hdfs.DFSClient: Slow ReadProcessor read fields took 478ms (threshold=20ms); ack: seqno: 3458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 477480817, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:33,800 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hdfs.DFSClient: Slow waitForAckedSeqno took 479ms (threshold=20ms) 2015-10-16 13:01:33,815 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/e535e906c69446f1990f9d3eb6cb257a as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e535e906c69446f1990f9d3eb6cb257a 2015-10-16 13:01:33,823 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:01:33,826 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:01:33,835 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ae383411b03f466a8a9c579dfd503a31, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ae383411b03f466a8a9c579dfd503a31 2015-10-16 13:01:33,843 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3448acfb80904a5daab336cb049ae08c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3448acfb80904a5daab336cb049ae08c 2015-10-16 13:01:33,877 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/30a2056e502449f2b17df78d05b011db, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/30a2056e502449f2b17df78d05b011db 2015-10-16 13:01:33,877 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into e535e906c69446f1990f9d3eb6cb257a(size=212.5 M), total size for store is 13.3 G. This selection was in queue for 0sec, and took 3sec to execute. 2015-10-16 13:01:33,877 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=2, time=36903599313007953; duration=3sec 2015-10-16 13:01:33,877 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:01:34,274 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630333_1099518554963] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 22582 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 710046, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:01:34,274 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 32004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 579817, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:34,274 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:01:34,274 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:01:34,274 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:01:34,274 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:01:34,274 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:01:34,579 DEBUG [PriorityRpcServer.handler=7,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:01:34,579 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134284656 > lower bound=16777216 2015-10-16 13:01:34,579 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:01:34,609 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 33589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 707156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:34,798 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630335_1099518554965] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 203 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 837411, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:34,798 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630334_1099518554964] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 23253 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:01:34,798 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 34377 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 586858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:34,798 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:34,798 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:34,798 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:34,798 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:34,798 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:34,978 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 34688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 754927, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:35,286 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 36054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 684074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:35,286 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630334_1099518554964] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 23870 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801690, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:01:35,286 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:35,286 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630335_1099518554965] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 903 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 958274, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:35,286 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:35,286 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:35,286 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:35,286 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:35,452 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=8989885, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ca755e0ac9844f7186855ea59073c583 2015-10-16 13:01:35,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ca755e0ac9844f7186855ea59073c583 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ca755e0ac9844f7186855ea59073c583 2015-10-16 13:01:35,471 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ca755e0ac9844f7186855ea59073c583, entries=466280, sequenceid=8989885, filesize=70.8 M 2015-10-16 13:01:35,473 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134284240, currentsize=18.96 MB/19876960 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 893ms, sequenceid=8989885, compaction requested=true 2015-10-16 13:01:35,473 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:01:35,473 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 4 compacting, 2 eligible, 10 blocking 2015-10-16 13:01:35,474 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:01:35,474 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:01:35,474 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:01:35,749 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:01:35,796 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630334_1099518554964] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 24496 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 778460, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:01:35,796 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630326_1099518554956] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 38521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 583340, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:35,796 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:35,796 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:35,796 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:35,796 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:35,796 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:35,802 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:35,819 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025687340 with entries=77867, filesize=122.06 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025695750 2015-10-16 13:01:35,821 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025679351 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025679351 2015-10-16 13:01:36,338 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630336_1099518554966] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 2561 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 409370, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:36,338 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630337_1099518554967] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 25205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:36,339 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:36,339 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:36,339 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:36,457 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#41 average throughput is 73.35 MB/sec, slept 0 time(s) and total slept time is 0 ms. 0 active compactions remaining, total limit is unlimited 2015-10-16 13:01:36,475 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b0b6dc71140942608d43144b078b55be as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b0b6dc71140942608d43144b078b55be 2015-10-16 13:01:36,483 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:01:36,486 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:01:36,493 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/feb37f4f73a84697a2c896c9c77546e0, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/feb37f4f73a84697a2c896c9c77546e0 2015-10-16 13:01:36,502 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1a8ce525811e466586be9d44ce7aa809, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1a8ce525811e466586be9d44ce7aa809 2015-10-16 13:01:36,510 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/860e1764c9fc4cfabb66b883cb15b410, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/860e1764c9fc4cfabb66b883cb15b410 2015-10-16 13:01:36,518 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6ddaf1fa9a24403ca59be04a8a37a783, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6ddaf1fa9a24403ca59be04a8a37a783 2015-10-16 13:01:36,519 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into b0b6dc71140942608d43144b078b55be(size=1.5 G), total size for store is 13.3 G. This selection was in queue for 0sec, and took 20sec to execute. 2015-10-16 13:01:36,519 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=1.5 G, priority=4, time=36903584630732046; duration=20sec 2015-10-16 13:01:36,519 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:01:37,310 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630336_1099518554966] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 7503 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 421932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:37,310 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:37,310 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:37,310 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:37,310 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:37,310 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:38,339 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630336_1099518554966] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 13127 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 445475, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:38,339 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:01:38,339 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:01:38,339 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:01:38,339 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:01:38,339 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:01:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.2857142857142857, tune compaction throughput to 12.86 MB/sec 2015-10-16 13:01:39,264 DEBUG [PriorityRpcServer.handler=5,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:01:39,264 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134276656 > lower bound=16777216 2015-10-16 13:01:39,293 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.86 MB 2015-10-16 13:01:39,324 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630336_1099518554966] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 698754, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:39,379 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630336_1099518554966] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 18659 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 493898, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:39,379 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:01:39,379 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:39,380 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:01:39,379 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:01:40,170 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630336_1099518554966] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 22904 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 480335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:40,170 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630338_1099518554968] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 698679, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:01:40,171 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:01:40,171 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:01:40,171 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:01:40,171 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:01:40,171 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:01:40,410 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630336_1099518554966] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 23556 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 886306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:40,485 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=9036826, memsize=128.9 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/171a0576faf346e8bc0e3ed06ac313d5 2015-10-16 13:01:40,490 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/171a0576faf346e8bc0e3ed06ac313d5 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/171a0576faf346e8bc0e3ed06ac313d5 2015-10-16 13:01:40,496 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/171a0576faf346e8bc0e3ed06ac313d5, entries=469360, sequenceid=9036826, filesize=71.3 M 2015-10-16 13:01:40,498 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.91 MB/135171920, currentsize=30.22 MB/31690320 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1205ms, sequenceid=9036826, compaction requested=true 2015-10-16 13:01:40,498 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:01:40,498 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 6 store files, 0 compacting, 6 eligible, 10 blocking 2015-10-16 13:01:40,501 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 10 permutations with 0 in ratio 2015-10-16 13:01:40,501 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:01:40,501 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:01:40,805 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 13:01:41,156 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630336_1099518554966] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 27467 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 463600, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:41,156 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:41,156 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:41,156 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:41,156 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:41,156 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:42,149 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630336_1099518554966] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 33044 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 575071, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:42,149 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:42,149 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:42,149 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:42,149 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:42,149 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:43,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630336_1099518554966] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 38599 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 435338, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:01:43,173 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:43,173 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:43,238 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:01:43,277 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025695750 with entries=77801, filesize=121.96 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025703238 2015-10-16 13:01:43,278 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025687340 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025687340 2015-10-16 13:01:43,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630339_1099518554969] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1079906, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:43,745 DEBUG [PriorityRpcServer.handler=11,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:01:43,745 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134280128 > lower bound=16777216 2015-10-16 13:01:43,746 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:01:43,770 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630339_1099518554969] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 2834 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 430389, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:44,060 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630339_1099518554969] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 4231 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 522513, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:44,060 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630340_1099518554970] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 359 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787520, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:01:44,060 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:01:44,060 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:01:44,060 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:01:44,060 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:01:44,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=9083460, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5bf68be7753346f1b219d8fdaed4654f 2015-10-16 13:01:44,598 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5bf68be7753346f1b219d8fdaed4654f as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5bf68be7753346f1b219d8fdaed4654f 2015-10-16 13:01:44,604 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5bf68be7753346f1b219d8fdaed4654f, entries=466300, sequenceid=9083460, filesize=70.8 M 2015-10-16 13:01:44,605 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134290080, currentsize=24.31 MB/25487120 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 860ms, sequenceid=9083460, compaction requested=true 2015-10-16 13:01:44,605 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:01:44,606 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 0 compacting, 7 eligible, 10 blocking 2015-10-16 13:01:44,608 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 446144011 starting at candidate #3 after considering 15 permutations with 2 in ratio 2015-10-16 13:01:44,608 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:01:44,608 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:01:44,608 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=425.5 M 2015-10-16 13:01:44,608 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e535e906c69446f1990f9d3eb6cb257a, keycount=1398750, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=8943252 2015-10-16 13:01:44,608 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ca755e0ac9844f7186855ea59073c583, keycount=466280, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=8989885 2015-10-16 13:01:44,609 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/171a0576faf346e8bc0e3ed06ac313d5, keycount=469360, bloomtype=ROW, size=71.3 M, encoding=NONE, seqNum=9036826 2015-10-16 13:01:44,609 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5bf68be7753346f1b219d8fdaed4654f, keycount=466300, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=9083460 2015-10-16 13:01:44,633 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:01:44,752 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630341_1099518554971] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 93 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 781360, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:44,752 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630339_1099518554969] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 7758 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 574333, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:44,752 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:44,752 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:44,752 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:44,752 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:44,752 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:45,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630339_1099518554969] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 10661 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910316, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:45,422 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630341_1099518554971] hdfs.DFSClient: Slow ReadProcessor read fields took 610ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1134969, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:45,709 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630339_1099518554969] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 12315 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 479823, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:45,709 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:45,709 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:45,709 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:45,709 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:45,709 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:46,199 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630341_1099518554971] hdfs.DFSClient: Slow ReadProcessor read fields took 647ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1212754, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:46,616 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630339_1099518554969] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 17152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 502381, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:46,616 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:46,616 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:46,616 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:46,616 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:46,616 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:46,976 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630341_1099518554971] hdfs.DFSClient: Slow ReadProcessor read fields took 651ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1320581, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:47,527 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630339_1099518554969] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 22031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 458442, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:47,527 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:47,527 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:47,527 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:47,527 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:47,527 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:47,753 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630341_1099518554971] hdfs.DFSClient: Slow ReadProcessor read fields took 654ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1180258, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:48,281 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:48,281 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:48,281 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:48,311 DEBUG [PriorityRpcServer.handler=15,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:01:48,311 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134276992 > lower bound=16777216 2015-10-16 13:01:48,311 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 13:01:48,340 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630339_1099518554969] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 26246 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 617617, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:48,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630342_1099518554972] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 88 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 770522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:01:48,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630339_1099518554969] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 26656 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 609228, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:48,457 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:48,457 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:48,457 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:48,457 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:48,457 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:48,531 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630341_1099518554971] hdfs.DFSClient: Slow ReadProcessor read fields took 651ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1123483, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:49,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630339_1099518554969] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 29819 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 560056, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:49,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630342_1099518554972] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 966 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 743185, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:01:49,120 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 91ms (threshold=20ms) 2015-10-16 13:01:49,120 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 92ms (threshold=20ms) 2015-10-16 13:01:49,120 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 91ms (threshold=20ms) 2015-10-16 13:01:49,120 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 92ms (threshold=20ms) 2015-10-16 13:01:49,120 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 92ms (threshold=20ms) 2015-10-16 13:01:49,259 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=9130094, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/f651234a1b124afc8d9e7caa5808ae0a 2015-10-16 13:01:49,265 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/f651234a1b124afc8d9e7caa5808ae0a as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f651234a1b124afc8d9e7caa5808ae0a 2015-10-16 13:01:49,271 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f651234a1b124afc8d9e7caa5808ae0a, entries=466300, sequenceid=9130094, filesize=70.8 M 2015-10-16 13:01:49,272 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134290400, currentsize=24.48 MB/25668880 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 961ms, sequenceid=9130094, compaction requested=true 2015-10-16 13:01:49,273 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:01:49,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 4 compacting, 1 eligible, 10 blocking 2015-10-16 13:01:49,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:01:49,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:01:49,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:01:49,309 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630341_1099518554971] hdfs.DFSClient: Slow ReadProcessor read fields took 655ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1215226, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:49,983 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630339_1099518554969] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 34319 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 684993, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:49,983 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:49,983 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:49,983 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:49,983 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:49,983 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:50,086 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630341_1099518554971] hdfs.DFSClient: Slow ReadProcessor read fields took 659ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1297929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:50,371 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630339_1099518554969] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 35751 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844738, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:50,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630341_1099518554971] hdfs.DFSClient: Slow ReadProcessor read fields took 656ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1334246, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:50,969 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630339_1099518554969] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 38747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 551241, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:50,969 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630341_1099518554971] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 1433 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 720232, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:50,969 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:50,969 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:50,969 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:50,969 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:01:50,970 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:50,994 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:01:51,026 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025703238 with entries=77831, filesize=122.01 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025710994 2015-10-16 13:01:51,028 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025695750 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025695750 2015-10-16 13:01:51,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630341_1099518554971] hdfs.DFSClient: Slow ReadProcessor read fields took 619ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1201296, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:51,821 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630343_1099518554973] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 4341 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 465790, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:01:51,821 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:52,418 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630341_1099518554971] hdfs.DFSClient: Slow ReadProcessor read fields took 644ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1281983, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:52,750 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630343_1099518554973] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 9247 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 496547, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:01:52,750 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:01:52,750 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:01:52,750 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:01:52,750 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:01:52,751 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:01:52,992 DEBUG [PriorityRpcServer.handler=18,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:01:52,993 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134285280 > lower bound=16777216 2015-10-16 13:01:52,993 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 13:01:53,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630343_1099518554973] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 10623 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 897055, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:01:53,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630341_1099518554971] hdfs.DFSClient: Slow ReadProcessor read fields took 648ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:53,535 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630344_1099518554974] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 783316, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:01:53,535 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630343_1099518554973] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 13097 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 703141, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:01:53,535 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:53,535 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:53,535 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:53,535 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:53,535 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:01:53,972 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630341_1099518554971] hdfs.DFSClient: Slow ReadProcessor read fields took 646ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1221804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:01:54,042 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=9176728, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/6d8a6512bdaf491ebf74772c464cba92 2015-10-16 13:01:54,048 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/6d8a6512bdaf491ebf74772c464cba92 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6d8a6512bdaf491ebf74772c464cba92 2015-10-16 13:01:54,062 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6d8a6512bdaf491ebf74772c464cba92, entries=466300, sequenceid=9176728, filesize=70.8 M 2015-10-16 13:01:54,064 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134291200, currentsize=29.58 MB/31016720 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1071ms, sequenceid=9176728, compaction requested=true 2015-10-16 13:01:54,064 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:01:54,065 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 4 compacting, 2 eligible, 10 blocking 2015-10-16 13:01:54,065 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:01:54,065 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:01:54,065 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:01:54,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630343_1099518554973] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 16737 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 608791, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:01:54,212 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:01:54,212 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:54,212 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:54,212 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:54,212 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:01:54,749 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630345_1099518554975] hdfs.DFSClient: Slow ReadProcessor read fields took 648ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1231947, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:55,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630343_1099518554973] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 20783 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 658111, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:01:55,218 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630343_1099518554973] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 21582 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 423971, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:01:55,218 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:55,218 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:55,218 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:55,218 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:55,218 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:55,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630345_1099518554975] hdfs.DFSClient: Slow ReadProcessor read fields took 659ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1236896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:56,110 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630343_1099518554973] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 26200 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 723260, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:01:56,191 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630343_1099518554973] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 26465 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 413671, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:01:56,191 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:56,192 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:56,192 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:56,192 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:56,192 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:56,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630345_1099518554975] hdfs.DFSClient: Slow ReadProcessor read fields took 650ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1171261, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:57,099 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630343_1099518554973] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 31336 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 439037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:01:57,099 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:57,099 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:57,099 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:57,099 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:57,099 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:01:57,101 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630345_1099518554975] hdfs.DFSClient: Slow ReadProcessor read fields took 671ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1372106, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:57,567 DEBUG [PriorityRpcServer.handler=19,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:01:57,567 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134264656 > lower bound=16777216 2015-10-16 13:01:57,567 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 13:01:57,593 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630343_1099518554973] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 34042 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 514227, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:01:57,900 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630346_1099518554976] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 373 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692801, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:01:57,900 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630343_1099518554973] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 35445 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 521517, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:01:57,900 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:57,900 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:57,901 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:57,901 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:57,901 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:01:57,902 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630345_1099518554975] hdfs.DFSClient: Slow ReadProcessor read fields took 684ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1457167, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:58,509 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=9223358, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/4773e056415e476ca810c569c7b46a0a 2015-10-16 13:01:58,550 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630343_1099518554973] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 38812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 475094, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:01:58,550 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:58,550 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:58,550 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:01:58,550 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:58,550 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:01:58,551 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/4773e056415e476ca810c569c7b46a0a as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4773e056415e476ca810c569c7b46a0a 2015-10-16 13:01:58,562 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4773e056415e476ca810c569c7b46a0a, entries=466260, sequenceid=9223358, filesize=70.8 M 2015-10-16 13:01:58,563 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134278640, currentsize=26.77 MB/28070400 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 996ms, sequenceid=9223358, compaction requested=true 2015-10-16 13:01:58,563 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:01:58,563 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 4 compacting, 3 eligible, 10 blocking 2015-10-16 13:01:58,564 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222836300 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 13:01:58,564 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:01:58,564 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:01:58,565 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 13:01:58,565 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f651234a1b124afc8d9e7caa5808ae0a, keycount=466300, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=9130094 2015-10-16 13:01:58,565 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6d8a6512bdaf491ebf74772c464cba92, keycount=466300, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=9176728 2015-10-16 13:01:58,565 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4773e056415e476ca810c569c7b46a0a, keycount=466260, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=9223358 2015-10-16 13:01:58,579 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:01:58,579 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:01:58,679 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630345_1099518554975] hdfs.DFSClient: Slow ReadProcessor read fields took 651ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 987712, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:01:58,772 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 179ms (threshold=20ms) 2015-10-16 13:01:58,775 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Slow sync cost: 182 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:01:58,793 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025710994 with entries=79705, filesize=124.94 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025718579 2015-10-16 13:01:58,794 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025703238 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025703238 2015-10-16 13:01:58,796 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630348_1099518554978] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907658, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:59,340 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:59,340 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630348_1099518554978] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 2891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 487789, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:01:59,340 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:59,340 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:01:59,341 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:01:59,584 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630348_1099518554978] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 3577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913289, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:00,140 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630347_1099518554977] hdfs.DFSClient: Slow ReadProcessor read fields took 1429ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1278212, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:02:00,234 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630345_1099518554975] hdfs.DFSClient: Slow ReadProcessor read fields took 1434ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1338580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:00,333 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630345_1099518554975] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 3247 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 722003, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:00,333 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630348_1099518554978] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 7462 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 604388, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:00,333 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:00,333 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:00,333 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:00,333 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:01,313 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630348_1099518554978] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 12594 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 446461, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:01,314 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:01,314 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:01,314 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:01,314 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:01,314 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:02:01,696 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630347_1099518554977] hdfs.DFSClient: Slow ReadProcessor read fields took 1433ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1341831, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:02:01,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630345_1099518554975] hdfs.DFSClient: Slow ReadProcessor read fields took 1380ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983089, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:02,126 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630348_1099518554978] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 16792 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 493911, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:02,126 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:02,126 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:02,126 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:02,126 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:02,126 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:02,221 DEBUG [PriorityRpcServer.handler=13,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:02:02,222 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134298720 > lower bound=16777216 2015-10-16 13:02:02,222 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.03 MB 2015-10-16 13:02:02,479 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630349_1099518554979] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 319 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 24888674, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:02:02,900 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:02:02,900 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630348_1099518554978] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 20323 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 581965, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:02,900 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630349_1099518554979] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 1012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 727857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:02:02,900 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:02:02,900 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:02:02,900 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:02:02,900 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:02:03,000 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=9269996, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d501fca2d60342fbbe6796f418616e98 2015-10-16 13:02:03,005 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d501fca2d60342fbbe6796f418616e98 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d501fca2d60342fbbe6796f418616e98 2015-10-16 13:02:03,011 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d501fca2d60342fbbe6796f418616e98, entries=466340, sequenceid=9269996, filesize=70.8 M 2015-10-16 13:02:03,013 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134301760, currentsize=20.11 MB/21086480 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 791ms, sequenceid=9269996, compaction requested=true 2015-10-16 13:02:03,013 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:02:03,252 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630347_1099518554977] hdfs.DFSClient: Slow ReadProcessor read fields took 1443ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1158659, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:02:03,345 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630345_1099518554975] hdfs.DFSClient: Slow ReadProcessor read fields took 1424ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1553392, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:03,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630348_1099518554978] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 24228 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 460978, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:03,641 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:02:03,641 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:02:03,641 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:02:03,641 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:02:03,641 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:02:03,966 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630348_1099518554978] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 25458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 708176, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:04,797 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630348_1099518554978] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 29812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 512523, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:04,797 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:02:04,797 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:02:04,797 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:02:04,807 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630347_1099518554977] hdfs.DFSClient: Slow ReadProcessor read fields took 1434ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1389077, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:02:04,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630345_1099518554975] hdfs.DFSClient: Slow ReadProcessor read fields took 1424ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1063701, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:05,611 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630348_1099518554978] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 33989 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489912, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:05,611 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:02:05,611 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:02:05,611 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:02:05,611 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:02:05,611 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:02:06,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630347_1099518554977] hdfs.DFSClient: Slow ReadProcessor read fields took 1433ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174651, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:02:06,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630345_1099518554975] hdfs.DFSClient: Slow ReadProcessor read fields took 1426ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1143554, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:06,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630345_1099518554975] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 3887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 643614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:06,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630348_1099518554978] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 38742 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 470565, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:06,530 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:06,530 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:06,530 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:02:06,530 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:06,531 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:06,563 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:02:06,601 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025718579 with entries=77784, filesize=121.94 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025726563 2015-10-16 13:02:06,603 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025710994 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025710994 2015-10-16 13:02:06,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630350_1099518554980] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043456, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:06,899 DEBUG [PriorityRpcServer.handler=19,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:02:06,899 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 13:02:06,899 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 13:02:07,453 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630350_1099518554980] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 4840 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 379582, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:07,453 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:07,453 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:07,453 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:07,453 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:07,453 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:07,917 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630347_1099518554977] hdfs.DFSClient: Slow ReadProcessor read fields took 1430ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1243335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:02:08,009 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630345_1099518554975] hdfs.DFSClient: Slow ReadProcessor read fields took 1385ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1121866, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:08,040 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630350_1099518554980] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 7589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 628952, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:08,323 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630350_1099518554980] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 8967 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 456336, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:08,323 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:08,323 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:08,323 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:08,323 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:08,323 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:08,404 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630350_1099518554980] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 9066 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 60190291, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:08,405 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 60ms (threshold=20ms) 2015-10-16 13:02:08,405 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 60ms (threshold=20ms) 2015-10-16 13:02:08,405 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 60ms (threshold=20ms) 2015-10-16 13:02:08,405 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 60ms (threshold=20ms) 2015-10-16 13:02:08,405 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 60ms (threshold=20ms) 2015-10-16 13:02:09,339 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630350_1099518554980] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 14460 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 534429, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:09,339 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:02:09,339 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:02:09,339 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:02:09,339 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:02:09,472 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630347_1099518554977] hdfs.DFSClient: Slow ReadProcessor read fields took 1440ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1240044, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:02:09,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630351_1099518554981] hdfs.DFSClient: Slow ReadProcessor read fields took 1402ms (threshold=20ms); ack: seqno: 4182 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1130377, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:10,109 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630350_1099518554980] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 18748 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 426361, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:10,109 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:10,109 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:02:10,109 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:02:10,109 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:11,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630347_1099518554977] hdfs.DFSClient: Slow ReadProcessor read fields took 1442ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1181596, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:02:11,069 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630350_1099518554980] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 24354 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 488498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:11,069 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630347_1099518554977] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 1336 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7771814, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:02:11,069 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:02:11,069 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:11,069 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:11,070 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:11,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630351_1099518554981] hdfs.DFSClient: Slow ReadProcessor read fields took 1405ms (threshold=20ms); ack: seqno: 4348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1168245, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:11,818 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630350_1099518554980] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 28619 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 418076, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:11,818 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:11,818 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:12,582 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630347_1099518554977] hdfs.DFSClient: Slow ReadProcessor read fields took 1388ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1132633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:02:12,674 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630351_1099518554981] hdfs.DFSClient: Slow ReadProcessor read fields took 1417ms (threshold=20ms); ack: seqno: 4517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1218915, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:12,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630351_1099518554981] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 4560 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751620, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:12,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630347_1099518554977] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 1665 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 722165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:02:12,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630350_1099518554980] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 33299 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1030042, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:13,636 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630350_1099518554980] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 38377 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 396632, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:13,636 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:13,636 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:13,636 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:13,637 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:13,769 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630350_1099518554980] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 38971 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 567080, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:13,864 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:02:14,138 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630347_1099518554977] hdfs.DFSClient: Slow ReadProcessor read fields took 1393ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1269938, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:02:14,229 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630351_1099518554981] hdfs.DFSClient: Slow ReadProcessor read fields took 1399ms (threshold=20ms); ack: seqno: 4684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1465597, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:14,439 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630350_1099518554980] hdfs.DFSClient: Slow ReadProcessor read fields took 546ms (threshold=20ms); ack: seqno: 39664 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1892861, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:14,439 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 546ms (threshold=20ms) 2015-10-16 13:02:14,442 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025726563 with entries=77891, filesize=122.11 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025733864 2015-10-16 13:02:14,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 556ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1283564, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:14,987 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 2879 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692220, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:14,988 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:02:14,988 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:02:14,988 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:02:14,988 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:02:14,988 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:02:15,693 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630347_1099518554977] hdfs.DFSClient: Slow ReadProcessor read fields took 1442ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1181794, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:02:15,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630351_1099518554981] hdfs.DFSClient: Slow ReadProcessor read fields took 1444ms (threshold=20ms); ack: seqno: 4851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1271394, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:15,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 7581 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 622020, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:15,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630351_1099518554981] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 4910 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794509, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:15,872 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:02:15,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630347_1099518554977] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 1982 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 763895, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:02:15,872 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:02:15,872 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:02:15,872 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:02:15,872 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:02:16,854 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 12675 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 466205, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:16,854 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:16,854 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:16,854 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:16,854 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:16,854 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:17,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630347_1099518554977] hdfs.DFSClient: Slow ReadProcessor read fields took 1359ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1228123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:02:17,339 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630351_1099518554981] hdfs.DFSClient: Slow ReadProcessor read fields took 1384ms (threshold=20ms); ack: seqno: 5018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1080182, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:17,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 16827 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 472101, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:17,668 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:02:17,668 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:02:17,668 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:02:17,668 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:02:17,668 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:02:18,653 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 22446 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 456415, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:18,653 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:18,653 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:18,653 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:18,653 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:18,802 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630353_1099518554983] hdfs.DFSClient: Slow ReadProcessor read fields took 1401ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1104255, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:18,894 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630351_1099518554981] hdfs.DFSClient: Slow ReadProcessor read fields took 1433ms (threshold=20ms); ack: seqno: 5185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1086712, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:18,906 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 13:02:18,906 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 13:02:18,906 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 13:02:18,906 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 13:02:18,906 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 13:02:19,142 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 24442 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 682339, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:19,573 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 26687 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 396583, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:19,573 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:02:19,573 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:02:19,573 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:02:19,573 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:02:20,391 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630353_1099518554983] hdfs.DFSClient: Slow ReadProcessor read fields took 1441ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 35140501, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:20,489 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:20,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630353_1099518554983] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 2472 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 632165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:20,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 31764 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489553, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:20,489 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:20,489 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:20,489 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:20,489 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:20,491 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630351_1099518554981] hdfs.DFSClient: Slow ReadProcessor read fields took 1476ms (threshold=20ms); ack: seqno: 5352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1394148, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:21,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630353_1099518554983] hdfs.DFSClient: Slow ReadProcessor read fields took 1400ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1386384, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:22,046 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630351_1099518554981] hdfs.DFSClient: Slow ReadProcessor read fields took 1438ms (threshold=20ms); ack: seqno: 5521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1116175, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:23,468 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630353_1099518554983] hdfs.DFSClient: Slow ReadProcessor read fields took 1452ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1475309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:23,602 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630351_1099518554981] hdfs.DFSClient: Slow ReadProcessor read fields took 1433ms (threshold=20ms); ack: seqno: 5688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188749, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:25,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630353_1099518554983] hdfs.DFSClient: Slow ReadProcessor read fields took 1455ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1386513, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:25,117 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630353_1099518554983] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 2929 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 739813, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:25,157 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630351_1099518554981] hdfs.DFSClient: Slow ReadProcessor read fields took 1452ms (threshold=20ms); ack: seqno: 5855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1387874, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:26,579 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630353_1099518554983] hdfs.DFSClient: Slow ReadProcessor read fields took 1411ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1285269, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:26,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630351_1099518554981] hdfs.DFSClient: Slow ReadProcessor read fields took 1453ms (threshold=20ms); ack: seqno: 6022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1032249, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:28,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630353_1099518554983] hdfs.DFSClient: Slow ReadProcessor read fields took 1448ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1532810, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:28,268 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630351_1099518554981] hdfs.DFSClient: Slow ReadProcessor read fields took 1455ms (threshold=20ms); ack: seqno: 6189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1403487, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:29,695 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630353_1099518554983] hdfs.DFSClient: Slow ReadProcessor read fields took 1460ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7702194, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:02:29,754 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#44 average throughput is 6.63 MB/sec, slept 20 time(s) and total slept time is 28471 ms. 1 active compactions remaining, total limit is 12.86 MB/sec 2015-10-16 13:02:29,773 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/033fa9fcecaf4102a78064e2d3004d9a as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/033fa9fcecaf4102a78064e2d3004d9a 2015-10-16 13:02:29,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 9057ms (threshold=20ms); ack: seqno: 33120 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834900, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:29,787 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:02:29,791 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:02:29,798 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f651234a1b124afc8d9e7caa5808ae0a, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f651234a1b124afc8d9e7caa5808ae0a 2015-10-16 13:02:29,807 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6d8a6512bdaf491ebf74772c464cba92, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6d8a6512bdaf491ebf74772c464cba92 2015-10-16 13:02:29,815 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4773e056415e476ca810c569c7b46a0a, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4773e056415e476ca810c569c7b46a0a 2015-10-16 13:02:29,815 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 033fa9fcecaf4102a78064e2d3004d9a(size=212.5 M), total size for store is 13.7 G. This selection was in queue for 0sec, and took 31sec to execute. 2015-10-16 13:02:29,815 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=0, time=36903627418541777; duration=31sec 2015-10-16 13:02:29,815 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 13:02:29,816 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 4 compacting, 2 eligible, 10 blocking 2015-10-16 13:02:29,817 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:02:29,817 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:02:29,817 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:02:29,817 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 4 compacting, 2 eligible, 10 blocking 2015-10-16 13:02:29,818 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:02:29,818 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:02:29,818 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:02:29,822 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630354_1099518554984] hdfs.DFSClient: Slow ReadProcessor read fields took 1453ms (threshold=20ms); ack: seqno: 6358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1434597, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:02:30,302 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536919696 > lower bound=16777216 2015-10-16 13:02:30,303 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 13:02:30,406 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 619ms (threshold=20ms); ack: seqno: 33121 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1053214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:30,609 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630355_1099518554985] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 288 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 743212, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:02:30,610 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630354_1099518554984] hdfs.DFSClient: Slow ReadProcessor read fields took 686ms (threshold=20ms); ack: seqno: 6525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1317825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:02:31,387 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630354_1099518554984] hdfs.DFSClient: Slow ReadProcessor read fields took 666ms (threshold=20ms); ack: seqno: 6694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1071566, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:02:32,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630354_1099518554984] hdfs.DFSClient: Slow ReadProcessor read fields took 667ms (threshold=20ms); ack: seqno: 6861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1202555, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:02:32,207 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#43 average throughput is 8.70 MB/sec, slept 41 time(s) and total slept time is 42105 ms. 0 active compactions remaining, total limit is 12.86 MB/sec 2015-10-16 13:02:32,221 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/d7e1cbe9289b40f0919088b7a88c8df8 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d7e1cbe9289b40f0919088b7a88c8df8 2015-10-16 13:02:32,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 1840ms (threshold=20ms); ack: seqno: 33122 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:32,247 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:02:32,252 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:02:32,257 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e535e906c69446f1990f9d3eb6cb257a, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e535e906c69446f1990f9d3eb6cb257a 2015-10-16 13:02:32,265 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ca755e0ac9844f7186855ea59073c583, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ca755e0ac9844f7186855ea59073c583 2015-10-16 13:02:32,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/171a0576faf346e8bc0e3ed06ac313d5, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/171a0576faf346e8bc0e3ed06ac313d5 2015-10-16 13:02:32,282 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5bf68be7753346f1b219d8fdaed4654f, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5bf68be7753346f1b219d8fdaed4654f 2015-10-16 13:02:32,282 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into d7e1cbe9289b40f0919088b7a88c8df8(size=425.5 M), total size for store is 13.7 G. This selection was in queue for 0sec, and took 47sec to execute. 2015-10-16 13:02:32,282 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=425.5 M, priority=3, time=36903613462236286; duration=47sec 2015-10-16 13:02:32,282 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:02:33,433 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630356_1099518554986] hdfs.DFSClient: Slow ReadProcessor read fields took 168ms (threshold=20ms); ack: seqno: 4166 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 168218842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:02:33,682 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=9456437, memsize=512.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/6e2adcb451d6484db55010b93e2befa9 2015-10-16 13:02:33,688 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/6e2adcb451d6484db55010b93e2befa9 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6e2adcb451d6484db55010b93e2befa9 2015-10-16 13:02:33,693 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6e2adcb451d6484db55010b93e2befa9, entries=1864360, sequenceid=9456437, filesize=283.2 M 2015-10-16 13:02:33,695 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 1448ms (threshold=20ms); ack: seqno: 33123 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 651154, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:33,695 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536919280, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 3393ms, sequenceid=9456437, compaction requested=true 2015-10-16 13:02:33,695 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:02:33,695 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 0 compacting, 7 eligible, 10 blocking 2015-10-16 13:02:33,699 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 6 files of size 5857067195 starting at candidate #1 after considering 15 permutations with 3 in ratio 2015-10-16 13:02:33,699 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:02:33,699 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 0 compacting, 7 eligible, 10 blocking 2015-10-16 13:02:33,703 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 6 files of size 5857067195 starting at candidate #1 after considering 15 permutations with 3 in ratio 2015-10-16 13:02:33,704 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:02:33,704 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:02:33,704 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 6 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=5.5 G 2015-10-16 13:02:33,704 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1f08a83c576d4583bc9945047a56a4d7, keycount=19985490, bloomtype=ROW, size=3.0 G, encoding=NONE, seqNum=7777652 2015-10-16 13:02:33,704 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b0b6dc71140942608d43144b078b55be, keycount=10256440, bloomtype=ROW, size=1.5 G, encoding=NONE, seqNum=8803365 2015-10-16 13:02:33,704 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d7e1cbe9289b40f0919088b7a88c8df8, keycount=2800690, bloomtype=ROW, size=425.5 M, encoding=NONE, seqNum=9083460 2015-10-16 13:02:33,704 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/033fa9fcecaf4102a78064e2d3004d9a, keycount=1398860, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=9223358 2015-10-16 13:02:33,704 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d501fca2d60342fbbe6796f418616e98, keycount=466340, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=9269996 2015-10-16 13:02:33,704 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6e2adcb451d6484db55010b93e2befa9, keycount=1864360, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=9456437 2015-10-16 13:02:33,750 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:02:34,536 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 675ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1629064, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:34,600 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 172 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1406974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:35,313 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 594ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1396687, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:35,369 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 339 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1173299, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:36,090 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 619ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1547203, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:36,867 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 674ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1492407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:36,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 671 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1283754, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:37,645 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 629ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1509839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:38,421 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 672ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1541839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.5714285714285714, tune compaction throughput to 15.71 MB/sec 2015-10-16 13:02:38,877 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 5183ms (threshold=20ms); ack: seqno: 33124 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 893611, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:39,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 676ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 19256828, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:39,255 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 35105 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 600202, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:39,255 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 1170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 18956231, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:39,255 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:39,255 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:39,255 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:39,255 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:39,256 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:39,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 481ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1284871, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:39,874 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 1339 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1131704, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:39,978 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:02:40,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 1497 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 708076, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:40,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630352_1099518554982] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 39375 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 475648, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:40,027 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:02:40,027 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:02:40,027 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:02:40,048 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025733864 with entries=77842, filesize=122.02 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025759978 2015-10-16 13:02:40,050 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025718579 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025718579 2015-10-16 13:02:40,056 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025726563 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025726563 2015-10-16 13:02:40,471 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 440ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1277707, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:40,868 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630359_1099518554989] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 4759 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 469821, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:40,868 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:40,868 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:40,868 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:40,868 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:40,868 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:41,106 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 514ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1146678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:41,749 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630359_1099518554989] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 9607 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 426349, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:41,749 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:41,749 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:41,749 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:41,749 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:41,749 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:41,751 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 530ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1254935, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:42,387 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630358_1099518554988] hdfs.DFSClient: Slow ReadProcessor read fields took 512ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1385649, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:42,513 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 2108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3294891, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:42,513 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630359_1099518554989] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 14034 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 559804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:42,513 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:42,513 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:42,513 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:42,513 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:42,513 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:42,660 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630359_1099518554989] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 14193 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804829, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:43,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 473ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1419639, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:43,072 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 2177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111838, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:43,158 DEBUG [PriorityRpcServer.handler=4,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:02:43,158 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134262896 > lower bound=16777216 2015-10-16 13:02:43,159 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 13:02:43,182 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630359_1099518554989] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 17156 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 549963, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:43,392 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630359_1099518554989] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 18108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 560296, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:43,392 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630361_1099518554991] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 244 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 678416, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:43,392 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:02:43,392 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:43,392 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:43,392 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:43,392 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:02:43,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 473ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1029037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:44,000 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630361_1099518554991] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 1130 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 662719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:02:44,000 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630359_1099518554989] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 21203 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 621447, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:44,000 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:02:44,000 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:02:44,000 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:02:44,000 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:02:44,000 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:02:44,064 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=9503065, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/dcd8c61116ff4bde80169a169655ffb3 2015-10-16 13:02:44,070 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/dcd8c61116ff4bde80169a169655ffb3 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/dcd8c61116ff4bde80169a169655ffb3 2015-10-16 13:02:44,076 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/dcd8c61116ff4bde80169a169655ffb3, entries=466230, sequenceid=9503065, filesize=70.8 M 2015-10-16 13:02:44,077 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134271120, currentsize=25.16 MB/26385840 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 919ms, sequenceid=9503065, compaction requested=false 2015-10-16 13:02:44,295 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 506ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1076151, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:44,812 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:44,812 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:44,813 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630359_1099518554989] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 25978 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 633114, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:44,930 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 488ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:45,567 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 505ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1109347, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:45,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 2901 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984164, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:45,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630359_1099518554989] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 30602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 584930, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:45,659 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:45,659 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:45,659 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:02:45,659 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:45,659 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:45,691 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 2909 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1203302, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:46,202 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 434ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1150148, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:46,414 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630359_1099518554989] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 34985 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 414608, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:46,414 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:46,414 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:46,414 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:46,414 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:46,414 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:46,839 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 512ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1107758, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:47,077 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:02:47,140 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025759978 with entries=77956, filesize=122.21 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025767077 2015-10-16 13:02:47,141 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025733864 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025733864 2015-10-16 13:02:47,142 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630362_1099518554992] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842210, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:47,298 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630362_1099518554992] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 758 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 388603, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:47,299 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:47,299 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:47,299 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:47,299 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:47,299 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:47,407 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134281856 > lower bound=16777216 2015-10-16 13:02:47,407 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.03 MB 2015-10-16 13:02:47,407 DEBUG [PriorityRpcServer.handler=6,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:02:47,475 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 515ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1187691, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:47,614 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 3352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942078, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:47,773 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 3486 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966137, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:47,796 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630362_1099518554992] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 2924 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 561841, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:48,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630362_1099518554992] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 4099 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 573527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:48,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630363_1099518554993] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 794 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 826721, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:48,027 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:02:48,027 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:02:48,027 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:02:48,027 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:02:48,027 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:02:48,112 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 320ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1086575, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:48,175 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 3519 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992795, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:48,281 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=9549704, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/7254bda43d704c6db1af82e0003cb236 2015-10-16 13:02:48,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/7254bda43d704c6db1af82e0003cb236 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7254bda43d704c6db1af82e0003cb236 2015-10-16 13:02:48,292 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7254bda43d704c6db1af82e0003cb236, entries=466350, sequenceid=9549704, filesize=70.8 M 2015-10-16 13:02:48,296 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134304480, currentsize=22.32 MB/23402400 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 889ms, sequenceid=9549704, compaction requested=true 2015-10-16 13:02:48,296 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:02:48,296 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 6 compacting, 2 eligible, 10 blocking 2015-10-16 13:02:48,297 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:02:48,297 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:02:48,297 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:02:48,353 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 3642 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885940, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:48,776 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630362_1099518554992] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 8188 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 498824, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:48,776 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:48,776 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:48,776 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:48,776 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:48,776 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:48,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 400ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1189023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:49,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 3838 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 889928, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:49,427 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 406ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 8230484, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:49,468 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 3864 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 843594, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:49,505 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 3871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1039436, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:49,586 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630362_1099518554992] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 12880 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 424688, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:49,586 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:49,586 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:49,586 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:49,586 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:49,586 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:49,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 3877 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1036812, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:49,761 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 3885 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1122271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:49,811 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 3891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023059, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:49,839 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 3898 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 944156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:49,867 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 3906 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 877348, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:49,940 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 3923 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,123 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 3947 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1067889, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,158 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 3953 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974830, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,184 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 3960 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926374, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,233 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 3973 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,260 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 3980 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,287 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 3988 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 889116, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,314 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 3997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909779, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,351 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 4006 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021593, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,377 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 4014 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869486, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,424 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630362_1099518554992] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 17794 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 458362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:50,425 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:50,425 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:50,425 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:50,425 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:50,425 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:50,426 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 4022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1057963, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,550 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 4028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1053332, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 4035 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983386, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,605 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 4042 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1001414, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,631 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 4049 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934013, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,705 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 4070 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866910, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 4076 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 977310, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 4092 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912190, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,817 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 4101 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 906029, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,844 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 4109 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 832793, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:50,955 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 4115 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1138464, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:51,082 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630360_1099518554990] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 4121 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954086, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:02:51,219 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630362_1099518554992] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 22457 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 570174, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:51,219 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 4255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 643593, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:51,219 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:51,219 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:51,219 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:51,219 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:51,219 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:51,325 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 4266 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857028, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:51,440 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 4274 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863399, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:51,601 DEBUG [PriorityRpcServer.handler=10,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:02:51,601 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134265136 > lower bound=16777216 2015-10-16 13:02:51,601 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:02:51,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 165ms (threshold=20ms); ack: seqno: 4280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 893124, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:51,627 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630362_1099518554992] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 24869 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 576281, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:51,886 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 274ms (threshold=20ms); ack: seqno: 4287 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910765, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:51,986 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630362_1099518554992] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 26889 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 353584, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:51,986 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630365_1099518554995] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 818586, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:02:51,986 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:51,986 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:51,987 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:51,987 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:51,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 4293 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 925914, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:52,262 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 266ms (threshold=20ms); ack: seqno: 4300 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1076151, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:52,414 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=9596333, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/bfa2f2d1ab4f40098639c288dfba155c 2015-10-16 13:02:52,418 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/bfa2f2d1ab4f40098639c288dfba155c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bfa2f2d1ab4f40098639c288dfba155c 2015-10-16 13:02:52,442 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bfa2f2d1ab4f40098639c288dfba155c, entries=466250, sequenceid=9596333, filesize=70.8 M 2015-10-16 13:02:52,443 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134276240, currentsize=25.35 MB/26581680 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 842ms, sequenceid=9596333, compaction requested=true 2015-10-16 13:02:52,443 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:02:52,443 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 6 compacting, 3 eligible, 10 blocking 2015-10-16 13:02:52,444 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222831258 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 13:02:52,444 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:02:52,444 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:02:52,445 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 13:02:52,445 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/dcd8c61116ff4bde80169a169655ffb3, keycount=466230, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=9503065 2015-10-16 13:02:52,445 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7254bda43d704c6db1af82e0003cb236, keycount=466350, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=9549704 2015-10-16 13:02:52,445 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bfa2f2d1ab4f40098639c288dfba155c, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=9596333 2015-10-16 13:02:52,458 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:02:52,585 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630362_1099518554992] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 29561 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836236, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:52,675 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 308ms (threshold=20ms); ack: seqno: 4430 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1384266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:52,713 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 4434 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 955344, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:52,714 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630362_1099518554992] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 30144 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 551980, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:52,714 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:02:52,714 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:02:52,714 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:02:52,714 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:02:52,714 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:02:52,777 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 4438 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1004416, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:53,015 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 201ms (threshold=20ms); ack: seqno: 25 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 203223984, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:02:53,097 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 317ms (threshold=20ms); ack: seqno: 4443 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1274737, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:53,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 4450 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1204049, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:53,478 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 259ms (threshold=20ms); ack: seqno: 4456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1149081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:53,538 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630362_1099518554992] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 35206 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 444461, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:02:53,538 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:02:53,538 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:02:53,538 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:02:53,538 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:02:53,538 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:02:53,605 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 4463 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1212354, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:53,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 682ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1371173, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:02:53,948 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 340ms (threshold=20ms); ack: seqno: 4469 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1219335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:54,033 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 4477 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1146578, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:54,142 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:02:54,172 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025767077 with entries=77839, filesize=122.02 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025774142 2015-10-16 13:02:54,175 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025759978 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025759978 2015-10-16 13:02:54,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 189ms (threshold=20ms); ack: seqno: 4485 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907179, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:54,311 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630367_1099518554997] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 614 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 524691, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:02:54,311 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 4546 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:54,311 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:54,311 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:54,311 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:54,311 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:54,311 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:02:54,393 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 4552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898586, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:54,822 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 327ms (threshold=20ms); ack: seqno: 4673 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1097855, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:54,944 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 4682 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 15295554, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:55,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 1156ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1284323, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:02:55,115 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630367_1099518554997] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 5228 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 449381, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:02:55,115 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:55,116 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:55,116 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:55,117 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 421 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908030, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:02:55,525 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 580ms (threshold=20ms); ack: seqno: 4684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1118556, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:55,722 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 195ms (threshold=20ms); ack: seqno: 4688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1352337, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:55,745 DEBUG [PriorityRpcServer.handler=3,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:02:55,745 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134282016 > lower bound=16777216 2015-10-16 13:02:55,745 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:02:55,772 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630367_1099518554997] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 9226 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 519947, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:02:55,833 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 4695 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1092424, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:55,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 4745 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 725029, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:55,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630368_1099518554998] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 150 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 753775, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:55,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630367_1099518554997] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 9883 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 614140, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:02:55,938 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:55,938 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:55,938 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:55,938 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:55,938 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:02:56,185 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 208ms (threshold=20ms); ack: seqno: 4764 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1296197, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:56,283 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 1107ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1179381, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:02:56,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 270ms (threshold=20ms); ack: seqno: 4771 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1144554, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:56,509 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630367_1099518554997] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 13032 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 497626, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:02:56,509 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630368_1099518554998] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 974 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 732588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:02:56,509 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:02:56,509 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:02:56,509 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:02:56,509 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:02:56,509 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:02:56,588 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 4778 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1229843, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:56,630 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=9642969, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a16726e026c24746b9aba186624bb5f3 2015-10-16 13:02:56,635 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a16726e026c24746b9aba186624bb5f3 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a16726e026c24746b9aba186624bb5f3 2015-10-16 13:02:56,642 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a16726e026c24746b9aba186624bb5f3, entries=466320, sequenceid=9642969, filesize=70.8 M 2015-10-16 13:02:56,643 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134296000, currentsize=25.56 MB/26803280 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 898ms, sequenceid=9642969, compaction requested=false 2015-10-16 13:02:56,950 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 358ms (threshold=20ms); ack: seqno: 4784 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1158497, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:57,065 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 4791 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1319813, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:57,072 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630367_1099518554997] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 16026 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811627, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:02:57,209 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 4913 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:57,307 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:02:57,307 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:02:57,307 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 5009 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808911, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:57,307 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630367_1099518554997] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 17283 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 379017, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:02:57,307 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:02:57,307 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:02:57,307 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:02:57,400 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 5015 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932963, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:57,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 1135ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1267129, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:02:58,088 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:02:58,088 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:02:58,088 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630367_1099518554997] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 22122 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 431081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:02:58,088 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:58,088 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:02:58,393 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 991ms (threshold=20ms); ack: seqno: 5018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 13150870, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:58,486 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 5022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1483517, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:58,595 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 5028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1163327, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:58,715 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 5043 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921343, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:58,769 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 5050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863208, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:58,827 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 1149ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1130126, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:02:58,906 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:58,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630367_1099518554997] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 27290 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 358651, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:02:58,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 892 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 721729, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:02:58,907 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:02:58,907 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:02:58,907 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:02:58,907 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:02:58,908 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 134ms (threshold=20ms); ack: seqno: 5058 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1092593, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:59,026 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 5076 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038344, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:59,084 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 5100 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 893772, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:59,186 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 5106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 976471, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:59,287 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 5112 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983660, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:59,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 143ms (threshold=20ms); ack: seqno: 5119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1015275, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:59,484 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 5126 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934323, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:59,546 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 5133 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 846455, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:59,590 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 5142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926989, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:59,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630367_1099518554997] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 32004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 397211, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:02:59,689 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:02:59,689 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:02:59,689 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:02:59,689 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:02:59,689 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:02:59,771 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 150ms (threshold=20ms); ack: seqno: 5160 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1028767, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:59,797 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 5168 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 826761, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:02:59,820 DEBUG [PriorityRpcServer.handler=5,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:02:59,820 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 13:02:59,820 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:02:59,959 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 158ms (threshold=20ms); ack: seqno: 5175 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 935551, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:00,091 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 5182 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5387526, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:00,099 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 1113ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044493, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:00,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 5188 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942012, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:00,142 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 5194 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887661, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:00,169 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 5201 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 970775, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:00,205 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 5208 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 906080, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:00,239 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 5216 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:00,376 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 5225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006207, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:00,475 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630367_1099518554997] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 36598 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 415244, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:00,475 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:00,475 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:00,475 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:00,521 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 141ms (threshold=20ms); ack: seqno: 5233 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1112413, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:00,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 5241 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 996228, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:00,674 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 5249 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:00,907 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:03:00,955 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025774142 with entries=77901, filesize=122.12 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025780907 2015-10-16 13:03:00,957 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025767077 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025767077 2015-10-16 13:03:00,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630369_1099518554999] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940657, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:00,965 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 269ms (threshold=20ms); ack: seqno: 5268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1197735, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:01,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 274ms (threshold=20ms); ack: seqno: 5275 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1074229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:01,289 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630369_1099518554999] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 1211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804542, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:01,361 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 5352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895388, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:01,371 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 1147ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972683, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:01,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 1230 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 681635, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:01,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630369_1099518554999] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 1962 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 668797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:01,455 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:01,455 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:01,486 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 5357 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:01,595 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 5364 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937221, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:01,697 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 5370 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844564, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:01,811 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 5378 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 850858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:01,922 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 5386 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:02,026 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 5394 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 973397, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:02,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 5405 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911716, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:02,243 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 5420 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:02,356 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630369_1099518554999] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 6804 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 409030, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:02,356 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:03:02,356 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:03:02,356 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:02,356 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:02,356 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:02,357 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 5426 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862907, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:02,478 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 5441 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 824467, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:02,585 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 5447 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:02,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 1114ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050613, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:02,695 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 5454 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927673, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:02,814 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 5461 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922383, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:02,926 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 5469 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 817930, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:03,039 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 5477 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017261, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:03,186 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630369_1099518554999] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 11440 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 545946, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:03,186 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:03:03,186 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:03:03,186 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:03:03,186 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:03:03,186 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:03:03,187 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 5485 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905837, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:03,324 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 5495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964723, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:03,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 183ms (threshold=20ms); ack: seqno: 5502 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1123286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:03,546 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 5509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048736, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:03,682 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 5515 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1117123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:03,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 150ms (threshold=20ms); ack: seqno: 5521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 939112, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:03,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 1157ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005029, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:03,980 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 140ms (threshold=20ms); ack: seqno: 5528 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907496, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:04,005 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 5534 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:04,054 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630369_1099518554999] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 16304 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1015957, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:04,055 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:04,055 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:04,055 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:04,055 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:04,055 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:04,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 5542 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924047, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:04,346 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 207ms (threshold=20ms); ack: seqno: 5551 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1252427, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:04,388 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 5559 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825516, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:04,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 140ms (threshold=20ms); ack: seqno: 5568 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:04,660 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 5575 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 903130, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:04,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 165ms (threshold=20ms); ack: seqno: 5583 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 929494, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:04,975 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 140ms (threshold=20ms); ack: seqno: 5590 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 816267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:05,029 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 5595 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 817284, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:05,029 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630369_1099518554999] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 21474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 439667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:05,029 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:03:05,029 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:03:05,029 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:03:05,029 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:03:05,029 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:03:05,145 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 5602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 961868, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:05,187 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 1097ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072181, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:05,259 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 5610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1061998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:05,401 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 138ms (threshold=20ms); ack: seqno: 5617 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984631, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:05,580 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 154ms (threshold=20ms); ack: seqno: 5632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 894564, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:05,700 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630369_1099518554999] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 24667 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 793614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:05,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 142ms (threshold=20ms); ack: seqno: 5650 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870353, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:05,900 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 5666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 997540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:05,982 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630369_1099518554999] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 26070 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 482916, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:05,982 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:05,983 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:05,983 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:05,983 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:05,983 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:06,090 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 185ms (threshold=20ms); ack: seqno: 5673 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 998976, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:06,251 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 5696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888650, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:06,458 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 5827 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863377, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:06,460 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 1154ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1035197, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:06,576 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 5835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:06,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 5849 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 800211, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:06,762 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630369_1099518554999] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 30254 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 441179, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:06,762 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:06,762 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:06,763 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:07,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 718ms (threshold=20ms); ack: seqno: 5855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1133135, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:07,518 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 5858 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1366201, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:07,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 5866 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1096118, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:07,734 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:07,734 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:07,734 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:07,734 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:07,734 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:07,736 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 1157ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1091668, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:07,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 148ms (threshold=20ms); ack: seqno: 5873 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 944175, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:07,897 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 5886 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844166, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:07,949 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 5892 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813077, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:08,295 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 341ms (threshold=20ms); ack: seqno: 5901 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044558, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:08,307 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:03:08,341 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630366_1099518554996] hdfs.DFSClient: Slow ReadProcessor read fields took 550ms (threshold=20ms); ack: seqno: 2082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 548673711, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:08,342 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 13:03:08,372 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025780907 with entries=78163, filesize=122.53 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025788307 2015-10-16 13:03:08,373 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 5909 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1126308, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:08,374 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 880725, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:08,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 479 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 55552178, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:08,512 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 56ms (threshold=20ms) 2015-10-16 13:03:08,512 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 55ms (threshold=20ms) 2015-10-16 13:03:08,512 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 55ms (threshold=20ms) 2015-10-16 13:03:08,512 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 56ms (threshold=20ms) 2015-10-16 13:03:08,512 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 55ms (threshold=20ms) 2015-10-16 13:03:08,633 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 932 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 320897, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:08,634 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:08,634 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:08,634 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:08,634 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:08,634 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:08,690 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 312ms (threshold=20ms); ack: seqno: 5915 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1406018, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:08,793 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 5919 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1294194, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:09,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630371_1099518555001] hdfs.DFSClient: Slow ReadProcessor read fields took 628ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019977, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:09,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 325ms (threshold=20ms); ack: seqno: 5925 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1018340, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:09,239 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 5931 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995241, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:09,426 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 5701 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 433922, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:09,427 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:09,427 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:09,427 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:09,427 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:09,517 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 273ms (threshold=20ms); ack: seqno: 5939 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1269898, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:09,619 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 5947 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1242616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:09,897 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 274ms (threshold=20ms); ack: seqno: 5955 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1132380, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:10,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 5963 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1104308, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:10,279 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630371_1099518555001] hdfs.DFSClient: Slow ReadProcessor read fields took 1141ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1169118, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:10,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 271ms (threshold=20ms); ack: seqno: 5971 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1082010, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:10,352 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 10944 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 560640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:10,352 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630371_1099518555001] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 2382 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811695, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:10,353 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:10,353 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:10,353 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:10,353 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:10,353 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:10,411 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 5980 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1189155, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:10,704 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 292ms (threshold=20ms); ack: seqno: 5984 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1247242, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:10,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 5992 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1190703, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:10,884 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 13443 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914601, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:11,160 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 357ms (threshold=20ms); ack: seqno: 5999 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1461009, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:11,221 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 6006 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1276929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:11,308 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 6065 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 803319, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:11,308 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 15627 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 422479, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:11,308 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:03:11,308 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:03:11,308 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:03:11,308 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:03:11,308 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:03:11,421 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 6071 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012283, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:11,536 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 6078 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965268, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:11,552 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630371_1099518555001] hdfs.DFSClient: Slow ReadProcessor read fields took 1115ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1203317, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:11,661 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 6091 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085184, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:11,779 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 6097 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1204267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:11,996 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 213ms (threshold=20ms); ack: seqno: 6104 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1126261, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:12,113 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 6111 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188220, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:12,169 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 6119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 772619, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:12,169 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 20282 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 454928, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:12,169 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:03:12,169 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:03:12,169 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:03:12,169 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:03:12,169 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:03:12,294 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 6128 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1142450, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:12,314 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 13:03:12,416 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 6137 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1189357, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:12,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 6145 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 991921, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:12,729 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 190ms (threshold=20ms); ack: seqno: 6153 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1025923, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:12,824 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630371_1099518555001] hdfs.DFSClient: Slow ReadProcessor read fields took 1150ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1096635, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:12,844 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630371_1099518555001] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 2675 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 20836732, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:12,849 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 6166 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1326286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:12,972 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 6180 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1056962, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:13,072 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 6186 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 996139, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:13,109 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 6197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 852181, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:13,231 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 6205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1029050, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:13,342 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 6213 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 929690, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:13,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 6221 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 918464, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:13,592 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 6230 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 9285945, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:13,725 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 6241 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1106541, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:13,877 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630364_1099518554994] hdfs.DFSClient: Slow ReadProcessor read fields took 149ms (threshold=20ms); ack: seqno: 6246 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:14,006 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 6255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1312554, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:14,097 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630371_1099518555001] hdfs.DFSClient: Slow ReadProcessor read fields took 1152ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1269175, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:14,160 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 6269 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1419314, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:14,279 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 6275 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1198451, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:14,313 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 6283 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 991398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:14,428 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 6291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:14,553 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 6298 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1121456, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:14,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 6307 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1091170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:14,825 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 6315 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1267069, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:14,988 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 6330 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1164484, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:15,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 143ms (threshold=20ms); ack: seqno: 6343 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1190817, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:15,277 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 6349 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1084034, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:15,369 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630371_1099518555001] hdfs.DFSClient: Slow ReadProcessor read fields took 1167ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1676033, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:15,416 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 6356 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:15,562 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 6371 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1151706, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:15,688 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 6381 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1146301, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:15,822 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 6385 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1186192, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:15,950 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 6393 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1132208, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:15,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 6404 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 815228, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:16,141 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 142ms (threshold=20ms); ack: seqno: 6411 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1269494, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:16,268 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 6418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:16,395 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 6424 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1289727, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:16,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 6432 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1029472, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:16,642 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630371_1099518555001] hdfs.DFSClient: Slow ReadProcessor read fields took 1165ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1584563, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:16,668 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 153ms (threshold=20ms); ack: seqno: 6438 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1128088, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:16,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 6445 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1215145, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:16,923 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 6453 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1191537, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:17,059 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 6462 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1249316, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:17,214 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 6478 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102712, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:17,349 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 6487 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1197052, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:17,496 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 6500 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1053853, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:17,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 6506 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1189367, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:17,751 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 6513 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188465, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:17,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 6520 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1264482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:17,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630371_1099518555001] hdfs.DFSClient: Slow ReadProcessor read fields took 1171ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1553825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:17,980 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#46 average throughput is 8.10 MB/sec, slept 20 time(s) and total slept time is 21848 ms. 1 active compactions remaining, total limit is 15.71 MB/sec 2015-10-16 13:03:17,994 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/83f302aa6ba5460facf177537219b9c6 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/83f302aa6ba5460facf177537219b9c6 2015-10-16 13:03:18,009 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 5074ms (threshold=20ms); ack: seqno: 24601 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728692, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:18,010 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:03:18,014 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:03:18,020 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/dcd8c61116ff4bde80169a169655ffb3, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/dcd8c61116ff4bde80169a169655ffb3 2015-10-16 13:03:18,029 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7254bda43d704c6db1af82e0003cb236, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7254bda43d704c6db1af82e0003cb236 2015-10-16 13:03:18,037 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bfa2f2d1ab4f40098639c288dfba155c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bfa2f2d1ab4f40098639c288dfba155c 2015-10-16 13:03:18,037 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 83f302aa6ba5460facf177537219b9c6(size=212.5 M), total size for store is 14.3 G. This selection was in queue for 0sec, and took 25sec to execute. 2015-10-16 13:03:18,037 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=0, time=36903681298488807; duration=25sec 2015-10-16 13:03:18,037 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:03:18,037 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 6 compacting, 2 eligible, 10 blocking 2015-10-16 13:03:18,039 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:03:18,039 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:03:18,039 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:03:18,090 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 207ms (threshold=20ms); ack: seqno: 6527 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1420252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:18,392 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 297ms (threshold=20ms); ack: seqno: 6536 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1549921, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:18,513 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 6545 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1610666, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:18,723 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536923776 > lower bound=16777216 2015-10-16 13:03:18,724 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 13:03:18,814 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 297ms (threshold=20ms); ack: seqno: 6553 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1573495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:18,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 822ms (threshold=20ms); ack: seqno: 24602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 674367, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:18,969 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 149ms (threshold=20ms); ack: seqno: 6562 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 11794177, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:19,227 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 252ms (threshold=20ms); ack: seqno: 6569 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1297099, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:19,318 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630373_1099518555003] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 766 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 707412, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:03:19,345 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 6575 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1296909, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:19,654 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 305ms (threshold=20ms); ack: seqno: 6582 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1550539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:19,776 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 6588 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196469, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:20,060 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 279ms (threshold=20ms); ack: seqno: 6595 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1323093, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:20,181 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 6603 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1055451, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:20,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 275ms (threshold=20ms); ack: seqno: 6611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014625, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:20,578 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 6620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111870, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:20,867 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 284ms (threshold=20ms); ack: seqno: 6628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1110699, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:20,971 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630374_1099518555004] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 3593 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 708467, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:21,008 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 6638 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1316912, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:21,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 306ms (threshold=20ms); ack: seqno: 6647 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1259731, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:21,445 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 6654 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1485646, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:21,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=9829411, memsize=512.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/00a17e991bd24cc8bf041c9485c13502 2015-10-16 13:03:21,576 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/00a17e991bd24cc8bf041c9485c13502 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/00a17e991bd24cc8bf041c9485c13502 2015-10-16 13:03:21,582 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/00a17e991bd24cc8bf041c9485c13502, entries=1864370, sequenceid=9829411, filesize=283.2 M 2015-10-16 13:03:21,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 2752ms (threshold=20ms); ack: seqno: 24603 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729053, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:21,584 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536923360, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2861ms, sequenceid=9829411, compaction requested=true 2015-10-16 13:03:21,584 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:03:21,584 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 6 compacting, 3 eligible, 10 blocking 2015-10-16 13:03:21,586 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 594104710 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 13:03:21,586 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:03:21,586 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:03:21,586 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=566.6 M 2015-10-16 13:03:21,586 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/83f302aa6ba5460facf177537219b9c6, keycount=1398830, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=9596333 2015-10-16 13:03:21,586 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a16726e026c24746b9aba186624bb5f3, keycount=466320, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=9642969 2015-10-16 13:03:21,586 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/00a17e991bd24cc8bf041c9485c13502, keycount=1864370, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=9829411 2015-10-16 13:03:21,602 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:03:21,799 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 350ms (threshold=20ms); ack: seqno: 6662 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1202355, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:21,918 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 6668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1316739, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:22,204 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 282ms (threshold=20ms); ack: seqno: 6674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1418043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:22,331 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 6681 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1594432, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:22,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 349ms (threshold=20ms); ack: seqno: 6688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1533948, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:22,828 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 140ms (threshold=20ms); ack: seqno: 6696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1417074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:22,886 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630376_1099518555006] hdfs.DFSClient: Slow ReadProcessor read fields took 1170ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1083088, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:23,141 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 307ms (threshold=20ms); ack: seqno: 6705 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1248194, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:23,265 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 6714 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1252107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:23,611 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 337ms (threshold=20ms); ack: seqno: 6723 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1501374, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:23,772 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 156ms (threshold=20ms); ack: seqno: 6730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1416573, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:23,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 141ms (threshold=20ms); ack: seqno: 6738 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1617835, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:24,134 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 6896 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921278, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:24,157 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630376_1099518555006] hdfs.DFSClient: Slow ReadProcessor read fields took 1171ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1146802, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:24,437 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 299ms (threshold=20ms); ack: seqno: 6903 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1291150, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:24,548 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 6910 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986359, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:24,861 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 310ms (threshold=20ms); ack: seqno: 6917 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1216386, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:25,380 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 519ms (threshold=20ms); ack: seqno: 6918 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1846463, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:25,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630376_1099518555006] hdfs.DFSClient: Slow ReadProcessor read fields took 1173ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019177, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:25,486 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630376_1099518555006] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 550 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 726777, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:25,662 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 275ms (threshold=20ms); ack: seqno: 6946 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1126780, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:25,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 6955 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1323445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:26,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 280ms (threshold=20ms); ack: seqno: 6964 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1389030, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:26,150 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 6972 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1073986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:26,247 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 6979 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1027740, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:26,335 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 6986 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1078574, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:26,581 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 242ms (threshold=20ms); ack: seqno: 6993 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1315660, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:26,675 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 7000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1456825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:26,702 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630376_1099518555006] hdfs.DFSClient: Slow ReadProcessor read fields took 1146ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111457, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:26,898 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 220ms (threshold=20ms); ack: seqno: 7006 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1339998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:27,000 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 7013 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1351993, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:27,093 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 7020 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1335847, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:27,349 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 252ms (threshold=20ms); ack: seqno: 7028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1438115, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:27,452 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 7036 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1420546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:27,702 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 246ms (threshold=20ms); ack: seqno: 7044 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1193227, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:27,791 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 7054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1251035, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:27,951 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 157ms (threshold=20ms); ack: seqno: 7061 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:27,975 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630376_1099518555006] hdfs.DFSClient: Slow ReadProcessor read fields took 1170ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1269861, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:28,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 174ms (threshold=20ms); ack: seqno: 7068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072524, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:28,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 7074 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1234236, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:28,467 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 216ms (threshold=20ms); ack: seqno: 7081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1376777, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:28,570 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 7087 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1233163, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:28,898 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 324ms (threshold=20ms); ack: seqno: 7094 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1370129, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:28,954 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 7101 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1380733, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:29,117 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 7214 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1218543, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:29,201 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 7226 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1365275, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:29,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630376_1099518555006] hdfs.DFSClient: Slow ReadProcessor read fields took 1170ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1298315, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:29,434 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 230ms (threshold=20ms); ack: seqno: 7233 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1501588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:29,517 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 7239 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1303161, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:29,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 7246 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995264, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:29,845 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 237ms (threshold=20ms); ack: seqno: 7252 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1451103, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:29,926 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 7260 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1397961, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:30,510 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 585ms (threshold=20ms); ack: seqno: 7261 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1813928, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:30,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630376_1099518555006] hdfs.DFSClient: Slow ReadProcessor read fields took 1165ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1145292, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:30,617 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 7301 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 877440, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:30,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 7305 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026056, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:30,696 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 7308 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1053901, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:30,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 7315 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1335106, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:31,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 237ms (threshold=20ms); ack: seqno: 7321 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1317129, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:31,081 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 9498ms (threshold=20ms); ack: seqno: 24604 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 746054, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:31,094 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 7328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1182618, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:31,267 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 25436 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 370607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:31,267 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:31,267 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:31,267 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:31,267 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:31,267 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:31,269 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 171ms (threshold=20ms); ack: seqno: 7336 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1082044, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:31,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 143ms (threshold=20ms); ack: seqno: 7343 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:31,536 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 7351 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914436, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:31,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 200ms (threshold=20ms); ack: seqno: 7355 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940069, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:31,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630376_1099518555006] hdfs.DFSClient: Slow ReadProcessor read fields took 1162ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1150273, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:31,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 7364 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 938004, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:32,125 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:03:32,125 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:03:32,125 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 30133 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 474557, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:32,125 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:03:32,125 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:03:32,125 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:03:32,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 270ms (threshold=20ms); ack: seqno: 7372 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010102, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:32,254 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 7380 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919577, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:32,306 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 7387 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 960843, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:32,448 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 7478 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 805845, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:32,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 7488 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:32,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 214ms (threshold=20ms); ack: seqno: 7495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019100, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:32,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 7498 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1342927, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:32,983 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 35010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 419655, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:32,984 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:32,984 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:32,984 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:32,984 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:32,984 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:33,014 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 7529 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934019, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:33,064 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630376_1099518555006] hdfs.DFSClient: Slow ReadProcessor read fields took 1154ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1161032, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:33,741 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630370_1099518555000] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 39185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 465866, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:33,741 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:33,741 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:33,742 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:33,742 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:33,742 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:33,759 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:03:33,803 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025788307 with entries=77780, filesize=121.93 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025813760 2015-10-16 13:03:33,805 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025774142 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025774142 2015-10-16 13:03:33,807 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1496276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:33,812 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025780907 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025780907 2015-10-16 13:03:34,065 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 956 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 678999, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:34,286 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#45 1094915/3729520 (29.36%), rate=7782.27 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=15.71 MB/sec, activeCompactions=2] 2015-10-16 13:03:34,293 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 1110ms (threshold=20ms); ack: seqno: 7696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 6561294, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:34,336 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630376_1099518555006] hdfs.DFSClient: Slow ReadProcessor read fields took 1127ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014330, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:34,423 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#45 sleep 1134 ms because current throughput is 72.46 MB/sec, max allowed is 7.86 MB/sec, already slept 29 time(s) and total slept time is 15532 ms till now. 2015-10-16 13:03:34,574 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 3434 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 428868, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:34,575 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 97ms (threshold=20ms) 2015-10-16 13:03:34,575 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 98ms (threshold=20ms) 2015-10-16 13:03:34,575 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 98ms (threshold=20ms) 2015-10-16 13:03:34,575 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 98ms (threshold=20ms) 2015-10-16 13:03:34,575 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 98ms (threshold=20ms) 2015-10-16 13:03:34,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 3933 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 403699, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:34,693 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:34,693 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:34,693 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:34,693 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:34,693 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:35,449 DEBUG [PriorityRpcServer.handler=11,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:03:35,449 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134265024 > lower bound=16777216 2015-10-16 13:03:35,450 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:03:35,478 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 8797 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 550970, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:35,596 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 9270 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 431259, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:35,596 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:35,596 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630378_1099518555008] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 90 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 626002, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:35,596 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:35,596 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:35,596 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:35,596 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:35,598 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 1173ms (threshold=20ms); ack: seqno: 7863 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1078679, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:35,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630376_1099518555006] hdfs.DFSClient: Slow ReadProcessor read fields took 1145ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1092845, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:36,132 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630378_1099518555008] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 768 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 716656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:36,132 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 12153 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 411758, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:36,132 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:36,132 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:36,132 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:36,132 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:36,132 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:03:36,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=9876038, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/6895f76af0994bfe834af878b90b609c 2015-10-16 13:03:36,434 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/6895f76af0994bfe834af878b90b609c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6895f76af0994bfe834af878b90b609c 2015-10-16 13:03:36,439 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6895f76af0994bfe834af878b90b609c, entries=466230, sequenceid=9876038, filesize=70.8 M 2015-10-16 13:03:36,441 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134270080, currentsize=28.71 MB/30106800 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 992ms, sequenceid=9876038, compaction requested=false 2015-10-16 13:03:36,870 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 1132ms (threshold=20ms); ack: seqno: 8030 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1138430, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:36,880 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630376_1099518555006] hdfs.DFSClient: Slow ReadProcessor read fields took 1138ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1126660, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:36,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630376_1099518555006] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 2027 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 726127, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:36,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 16830 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 434612, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:36,932 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:36,932 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:36,932 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:36,932 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:36,932 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:36,934 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 8034 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1325196, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:37,696 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 21064 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 466493, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:37,696 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:37,696 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:37,696 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:37,696 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:38,141 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630372_1099518555002] hdfs.DFSClient: Slow ReadProcessor read fields took 1076ms (threshold=20ms); ack: seqno: 8199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1093082, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:38,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630379_1099518555009] hdfs.DFSClient: Slow ReadProcessor read fields took 1107ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842378, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:03:38,191 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 13:03:38,435 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 25270 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 429473, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:38,435 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:38,435 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:38,435 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:38,435 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:38,435 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:38,668 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 162ms (threshold=20ms); ack: seqno: 25731 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 162127001, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:38,668 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 162ms (threshold=20ms) 2015-10-16 13:03:38,668 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 163ms (threshold=20ms) 2015-10-16 13:03:38,668 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 162ms (threshold=20ms) 2015-10-16 13:03:38,668 INFO [sync.2] wal.FSHLog: Slow sync cost: 162 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:38,668 INFO [sync.4] wal.FSHLog: Slow sync cost: 161 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:38,668 INFO [sync.0] wal.FSHLog: Slow sync cost: 162 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:38,668 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 162ms (threshold=20ms) 2015-10-16 13:03:38,668 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 162ms (threshold=20ms) 2015-10-16 13:03:38,668 INFO [sync.3] wal.FSHLog: Slow sync cost: 162 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:38,669 INFO [sync.1] wal.FSHLog: Slow sync cost: 162 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 13:03:38,952 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 26703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 877352, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:39,414 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630380_1099518555010] hdfs.DFSClient: Slow ReadProcessor read fields took 1123ms (threshold=20ms); ack: seqno: 8369 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1227340, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:03:39,425 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630379_1099518555009] hdfs.DFSClient: Slow ReadProcessor read fields took 1147ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1133093, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:03:39,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630380_1099518555010] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 8374 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1190412, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:03:39,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630379_1099518555009] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 2464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 855682, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:03:39,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630380_1099518555010] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 8459 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 756576, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:03:39,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 30015 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 574339, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:39,571 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:03:39,571 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:03:39,571 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:03:39,571 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:03:39,571 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:03:39,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630379_1099518555009] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 2911 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 771309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:03:39,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 31834 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 682312, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:39,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630380_1099518555010] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 8934 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779021, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:03:39,967 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:39,967 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:39,967 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:39,967 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:39,967 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:40,047 DEBUG [PriorityRpcServer.handler=9,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:03:40,047 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 13:03:40,047 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:03:40,346 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 33664 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 592779, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:40,346 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630379_1099518555009] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 3366 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:03:40,346 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630380_1099518555010] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 9380 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 722676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:03:40,346 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:40,346 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:40,346 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:40,346 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:40,346 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:40,729 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630379_1099518555009] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 3803 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 773995, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:03:40,729 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 35505 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 515507, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:40,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630380_1099518555010] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 9843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 721271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:03:40,730 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:40,730 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:40,730 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:40,730 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:40,730 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:40,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 36142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787851, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:41,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630381_1099518555011] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 4285 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 612902, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:41,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 37128 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 448254, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:41,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630380_1099518555010] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 10331 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 744635, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:03:41,128 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:41,129 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:41,129 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:41,129 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:41,472 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:03:41,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630377_1099518555007] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 39092 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 634438, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:03:41,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630382_1099518555012] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 10766 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 721769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:03:41,513 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630381_1099518555011] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 4725 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 720656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:41,513 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:41,513 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:41,513 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:41,513 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:41,513 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:41,518 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:03:41,544 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025813760 with entries=77795, filesize=121.95 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025821472 2015-10-16 13:03:41,546 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025788307 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025788307 2015-10-16 13:03:41,547 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 818267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:41,900 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630382_1099518555012] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 11235 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:03:41,901 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 1678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 622805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:41,901 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:03:41,902 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630381_1099518555011] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 5178 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908649, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:42,034 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630382_1099518555012] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 11373 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930608, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:03:42,297 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 3688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 401461, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:42,297 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630381_1099518555011] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 5639 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 657012, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:42,297 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630382_1099518555012] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 11648 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 820000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:03:42,298 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:42,298 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:42,298 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:42,298 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:42,679 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630381_1099518555011] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 6076 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 669679, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:42,679 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630382_1099518555012] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 12066 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 810990, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:03:42,679 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 5686 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 400038, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:42,680 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:42,680 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:42,680 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:42,680 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:43,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630382_1099518555012] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 12475 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844270, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:03:43,066 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:43,066 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:43,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630384_1099518555014] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 6512 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 791148, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:43,066 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:43,066 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:43,066 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:43,460 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630384_1099518555014] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 6962 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 663850, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:43,460 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630385_1099518555015] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 12895 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1554288, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:43,460 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 9725 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 625531, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:43,460 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:03:43,460 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:43,460 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:03:43,460 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:03:43,460 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:43,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630385_1099518555015] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 13275 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1071983, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:43,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 11687 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 362948, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:43,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630384_1099518555014] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 7414 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 696239, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:43,850 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:43,850 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:43,850 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:43,850 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:43,850 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:44,053 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 12100 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 753682, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:44,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 150ms (threshold=20ms); ack: seqno: 12177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 149206547, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:44,215 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 150ms (threshold=20ms) 2015-10-16 13:03:44,215 INFO [sync.2] wal.FSHLog: Slow sync cost: 150 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:44,215 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 150ms (threshold=20ms) 2015-10-16 13:03:44,215 INFO [sync.1] wal.FSHLog: Slow sync cost: 150 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:44,216 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 151ms (threshold=20ms) 2015-10-16 13:03:44,216 INFO [sync.0] wal.FSHLog: Slow sync cost: 150 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:44,216 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 151ms (threshold=20ms) 2015-10-16 13:03:44,216 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 151ms (threshold=20ms) 2015-10-16 13:03:44,216 INFO [sync.3] wal.FSHLog: Slow sync cost: 150 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:44,217 INFO [sync.4] wal.FSHLog: Slow sync cost: 150 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:44,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630385_1099518555015] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 13884 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3461657, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:44,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630384_1099518555014] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 7976 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1468782, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:44,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 12526 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 544921, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:44,319 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:44,319 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:44,319 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:44,319 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:44,319 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:44,713 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630385_1099518555015] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 14322 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:44,713 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630386_1099518555016] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 8443 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 757428, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:03:44,713 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 14412 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012899, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:44,713 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:44,713 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:44,713 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:45,102 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630386_1099518555016] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 8897 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 753705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:03:45,102 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 16257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 431174, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:45,102 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630387_1099518555017] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 14763 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 733010, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:03:45,103 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:45,103 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:45,103 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:45,370 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#47 average throughput is 23.19 MB/sec, slept 14 time(s) and total slept time is 16128 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 13:03:45,382 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ed793e60b87549749e48c76c641adea0 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ed793e60b87549749e48c76c641adea0 2015-10-16 13:03:45,390 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:03:45,393 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:03:45,402 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/83f302aa6ba5460facf177537219b9c6, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/83f302aa6ba5460facf177537219b9c6 2015-10-16 13:03:45,410 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a16726e026c24746b9aba186624bb5f3, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a16726e026c24746b9aba186624bb5f3 2015-10-16 13:03:45,419 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/00a17e991bd24cc8bf041c9485c13502, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/00a17e991bd24cc8bf041c9485c13502 2015-10-16 13:03:45,419 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into ed793e60b87549749e48c76c641adea0(size=566.5 M), total size for store is 14.6 G. This selection was in queue for 0sec, and took 23sec to execute. 2015-10-16 13:03:45,419 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=566.6 M, priority=0, time=36903710439781192; duration=23sec 2015-10-16 13:03:45,419 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:03:45,419 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 6 compacting, 2 eligible, 10 blocking 2015-10-16 13:03:45,420 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:03:45,420 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:03:45,420 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:03:45,448 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=278196896 > lower bound=16777216 2015-10-16 13:03:45,449 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=265.26 MB 2015-10-16 13:03:45,505 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 18228 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 512190, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:45,546 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630387_1099518555017] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 15277 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 689958, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:03:45,546 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 18258 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 537305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:45,546 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:45,546 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:45,546 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:45,546 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:45,546 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:45,997 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630387_1099518555017] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 15778 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 731118, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:03:45,997 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630388_1099518555018] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 638 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 809894, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:45,997 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 20519 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 513066, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:45,997 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:45,997 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:03:45,998 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:03:45,998 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:03:45,998 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:03:46,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630388_1099518555018] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 1225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 784132, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:46,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630387_1099518555017] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 16311 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 706178, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:03:46,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 22816 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 471998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:46,457 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:03:46,457 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:46,457 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:03:46,457 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:46,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630389_1099518555019] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 16816 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782927, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:46,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630388_1099518555018] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 1785 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 742351, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:03:46,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 25198 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 470515, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:46,915 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:46,915 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:46,915 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:46,915 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:46,915 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:47,109 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630389_1099518555019] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 16970 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 822819, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:47,433 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 28037 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 511899, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:47,433 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630390_1099518555020] hdfs.DFSClient: Slow ReadProcessor read fields took 237ms (threshold=20ms); ack: seqno: 2097 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 215837986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:03:47,434 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:47,434 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630389_1099518555019] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 17308 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866334, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:47,434 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:47,434 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:47,434 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:47,434 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:47,577 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=9972647, memsize=265.3 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a3dbcc837c8948a296f0adba4173583f 2015-10-16 13:03:47,582 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a3dbcc837c8948a296f0adba4173583f as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a3dbcc837c8948a296f0adba4173583f 2015-10-16 13:03:47,588 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a3dbcc837c8948a296f0adba4173583f, entries=966040, sequenceid=9972647, filesize=146.8 M 2015-10-16 13:03:47,589 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~265.32 MB/278210880, currentsize=58.59 MB/61437120 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2141ms, sequenceid=9972647, compaction requested=true 2015-10-16 13:03:47,589 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:03:47,589 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 6 compacting, 3 eligible, 10 blocking 2015-10-16 13:03:47,590 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 13:03:47,590 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:03:47,590 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:03:47,922 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:47,922 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 30765 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 457993, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:47,922 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:47,922 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630389_1099518555019] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 17886 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 685446, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:47,922 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:47,923 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:48,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 33669 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 501649, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:48,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630389_1099518555019] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 18538 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 726091, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:03:48,447 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:48,447 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:48,447 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:48,447 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:48,447 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:48,930 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 35595 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 724316, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:49,062 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 36079 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 494265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:49,062 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:03:49,062 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:49,062 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:03:49,062 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:03:49,062 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630391_1099518555021] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 19305 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 703135, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:49,615 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630391_1099518555021] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 19984 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005447, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:49,615 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630383_1099518555013] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 38830 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 606494, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:49,615 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:49,615 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:49,615 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:49,615 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:49,616 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:49,639 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:03:49,677 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025821472 with entries=77813, filesize=121.98 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025829639 2015-10-16 13:03:49,680 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 918231, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:49,681 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025813760 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025813760 2015-10-16 13:03:50,162 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630391_1099518555021] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 20664 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 884814, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:50,162 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 2482 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 465730, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:50,162 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:50,162 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:50,162 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:50,162 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:50,162 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:50,174 DEBUG [PriorityRpcServer.handler=5,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:03:50,174 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134279616 > lower bound=16777216 2015-10-16 13:03:50,175 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 13:03:50,199 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 2544 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 626462, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:50,617 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 4592 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 510922, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:50,617 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630394_1099518555024] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 21213 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736310, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:50,617 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630393_1099518555023] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 579 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751551, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:03:50,617 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:03:50,618 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:50,618 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:50,618 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:50,650 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630394_1099518555024] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 21221 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 699675, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:50,985 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10019280, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/131f43a51fae46638d8d4f2139e8a3f3 2015-10-16 13:03:50,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/131f43a51fae46638d8d4f2139e8a3f3 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/131f43a51fae46638d8d4f2139e8a3f3 2015-10-16 13:03:50,996 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/131f43a51fae46638d8d4f2139e8a3f3, entries=466290, sequenceid=10019280, filesize=70.8 M 2015-10-16 13:03:50,997 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134287840, currentsize=22.24 MB/23321600 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 823ms, sequenceid=10019280, compaction requested=true 2015-10-16 13:03:50,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:03:50,997 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 6 compacting, 4 eligible, 10 blocking 2015-10-16 13:03:50,999 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 302433356 starting at candidate #1 after considering 3 permutations with 1 in ratio 2015-10-16 13:03:50,999 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:03:50,999 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:03:50,999 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=288.4 M 2015-10-16 13:03:50,999 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6895f76af0994bfe834af878b90b609c, keycount=466230, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=9876038 2015-10-16 13:03:50,999 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a3dbcc837c8948a296f0adba4173583f, keycount=966040, bloomtype=ROW, size=146.8 M, encoding=NONE, seqNum=9972647 2015-10-16 13:03:50,999 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/131f43a51fae46638d8d4f2139e8a3f3, keycount=466290, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=10019280 2015-10-16 13:03:51,013 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:03:51,067 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630394_1099518555024] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 21727 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 680474, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:51,067 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 6765 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 465184, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:51,067 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 13:03:51,067 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 13:03:51,067 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 13:03:51,067 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 13:03:51,067 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 13:03:51,509 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630395_1099518555025] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 265 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 831831, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:03:51,509 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630394_1099518555024] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 22279 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:51,509 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 9067 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 445486, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:51,509 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:51,509 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:51,509 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:51,509 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:51,510 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:51,901 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630394_1099518555024] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 22736 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992456, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:51,901 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 10964 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 638534, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:51,901 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630395_1099518555025] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 671800, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:03:51,901 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:51,901 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:03:51,901 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:51,901 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:52,290 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630395_1099518555025] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 612715, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:03:52,290 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630396_1099518555026] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 23167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 892034, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:52,290 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 12809 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 626357, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:52,291 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:52,291 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:52,291 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:52,291 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:52,291 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:52,681 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630396_1099518555026] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 23594 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 6799896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:52,681 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630395_1099518555025] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 1610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 10313439, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:03:52,681 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 14753 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 501323, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:52,681 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:52,681 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:52,681 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:52,681 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:53,067 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630395_1099518555025] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 2054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830338, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:03:53,067 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630396_1099518555026] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 24050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 636128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:53,067 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 16686 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 548151, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:53,067 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:53,067 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:03:53,067 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:53,067 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:53,068 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:03:53,475 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 18642 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 720670, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:53,475 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630397_1099518555027] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 2527 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1356593, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:03:53,475 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:53,475 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:53,475 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:53,475 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:53,477 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630396_1099518555026] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 24442 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1024244, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:53,524 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630396_1099518555026] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 24449 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937728, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:53,615 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 18753 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 666165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:53,678 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630396_1099518555026] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 24566 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 960515, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:53,853 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630396_1099518555026] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 24691 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975672, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:53,951 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630397_1099518555027] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 3118 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 641585, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:03:53,951 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630396_1099518555026] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 24764 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827663, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:53,951 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 20401 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 474482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:53,951 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:53,951 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:53,951 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:53,951 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:53,951 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:54,020 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630396_1099518555026] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 24771 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1107932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:54,202 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630396_1099518555026] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 24902 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 977965, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:03:54,413 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:54,413 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:54,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 22669 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 681827, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:54,413 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:54,413 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:03:54,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630397_1099518555027] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 3682 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 743878, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:03:54,413 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:54,415 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 25008 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1360063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:54,483 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 25015 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111782, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:54,672 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 25146 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985610, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:54,893 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 59ms (threshold=20ms) 2015-10-16 13:03:54,893 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 58ms (threshold=20ms) 2015-10-16 13:03:54,893 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630399_1099518555029] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 4191 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 20276747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:03:54,894 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 24840 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 647959, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:54,894 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 25277 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1136998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:54,972 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 25284 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054845, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:55,079 DEBUG [PriorityRpcServer.handler=9,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:03:55,080 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 13:03:55,080 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:03:55,150 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 25405 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 774473, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:55,226 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#48 average throughput is 66.66 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 13:03:55,240 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/6f9732ddcc084bb598a1a1af24cb69e5 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6f9732ddcc084bb598a1a1af24cb69e5 2015-10-16 13:03:55,248 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:03:55,251 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:03:55,260 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6895f76af0994bfe834af878b90b609c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6895f76af0994bfe834af878b90b609c 2015-10-16 13:03:55,268 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a3dbcc837c8948a296f0adba4173583f, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a3dbcc837c8948a296f0adba4173583f 2015-10-16 13:03:55,276 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/131f43a51fae46638d8d4f2139e8a3f3, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/131f43a51fae46638d8d4f2139e8a3f3 2015-10-16 13:03:55,277 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 6f9732ddcc084bb598a1a1af24cb69e5(size=288.4 M), total size for store is 14.9 G. This selection was in queue for 0sec, and took 4sec to execute. 2015-10-16 13:03:55,277 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=288.4 M, priority=-1, time=36903739853207275; duration=4sec 2015-10-16 13:03:55,277 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:03:55,277 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 6 compacting, 2 eligible, 10 blocking 2015-10-16 13:03:55,278 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:03:55,278 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:03:55,278 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:03:55,331 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 25528 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 998043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:55,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 27293 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 589007, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:55,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 25578 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 647066, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:55,399 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:03:55,399 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:03:55,399 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:03:55,399 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:03:55,515 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 25585 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037933, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:55,696 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 25705 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1078776, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:55,951 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 153ms (threshold=20ms); ack: seqno: 25838 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1075265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:55,980 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=162030736 > lower bound=16777216 2015-10-16 13:03:55,980 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=154.47 MB 2015-10-16 13:03:56,013 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 30627 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 605699, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:56,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630400_1099518555030] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 89 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1267797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:56,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 30954 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 605890, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:56,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 25989 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908516, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:56,111 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:56,111 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:56,111 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:56,111 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:56,111 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:56,160 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 25995 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:56,320 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 26122 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 980972, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:56,501 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 26231 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927002, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:56,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 33526 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 417562, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:56,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630400_1099518555030] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 858 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794062, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:03:56,643 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:56,643 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:56,643 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:56,643 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:56,643 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:56,679 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 26361 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 977052, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:56,861 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 26493 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979087, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:57,001 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10075547, memsize=154.5 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/eda6f40996a1498e9a28b55a0344caa8 2015-10-16 13:03:57,006 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/eda6f40996a1498e9a28b55a0344caa8 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eda6f40996a1498e9a28b55a0344caa8 2015-10-16 13:03:57,012 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eda6f40996a1498e9a28b55a0344caa8, entries=562620, sequenceid=10075547, filesize=85.5 M 2015-10-16 13:03:57,014 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~154.52 MB/162030320, currentsize=26.49 MB/27779360 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1034ms, sequenceid=10075547, compaction requested=true 2015-10-16 13:03:57,014 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:03:57,014 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 6 compacting, 3 eligible, 10 blocking 2015-10-16 13:03:57,016 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 13:03:57,016 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:03:57,016 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:03:57,036 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 26641 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 998509, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:57,189 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630398_1099518555028] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 26797 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:03:57,189 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630392_1099518555022] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 36199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 498884, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:03:57,189 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:03:57,190 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:03:57,190 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:03:57,190 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:03:57,683 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:03:57,718 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025829639 with entries=77871, filesize=122.07 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025837683 2015-10-16 13:03:57,721 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025821472 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025821472 2015-10-16 13:03:57,757 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630402_1099518555032] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994612, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:57,757 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630401_1099518555031] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 27509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 755566, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:03:57,757 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:57,757 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:57,757 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:57,757 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:57,757 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:03:58,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630402_1099518555032] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 1257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 643948, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:58,350 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630401_1099518555031] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 28287 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2718276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:03:58,350 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630402_1099518555032] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 2402 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 598237, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:58,350 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:58,350 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:58,350 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:58,350 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:58,350 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:58,918 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630401_1099518555031] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 28959 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 735230, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:03:58,918 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630402_1099518555032] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 5184 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 526292, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:58,919 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:58,919 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:03:58,919 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:58,919 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:03:58,919 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:03:59,459 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:59,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 29639 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 685606, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:03:59,459 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:59,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630402_1099518555032] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 7955 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 448267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:59,459 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:59,459 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:59,459 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:03:59,978 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630402_1099518555032] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 10791 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 502921, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:03:59,978 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 30280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762413, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:03:59,978 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:59,978 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:59,978 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:59,978 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:03:59,978 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:04:00,415 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 30676 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937741, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:00,604 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:04:00,604 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:04:00,604 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:04:00,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630402_1099518555032] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 14161 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 486662, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:04:00,604 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:04:00,604 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:04:00,634 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 30820 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972952, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:00,772 DEBUG [PriorityRpcServer.handler=5,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:04:00,772 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134278576 > lower bound=16777216 2015-10-16 13:04:00,772 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:04:00,797 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630402_1099518555032] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 15113 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 731843, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:04:00,860 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 30965 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921917, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:00,970 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 30974 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 938864, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:01,096 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 30992 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020577, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:01,205 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 31000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910396, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:01,289 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630402_1099518555032] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 17532 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 724399, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:04:01,289 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630404_1099518555034] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 707 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:04:01,289 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:04:01,289 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:04:01,289 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:04:01,289 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:04:01,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 31007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1078623, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:01,403 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 31013 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:01,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 31020 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 929631, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:01,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10122180, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/17fc5c151c3b434bb288ffe52b5657fa 2015-10-16 13:04:01,597 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/17fc5c151c3b434bb288ffe52b5657fa as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/17fc5c151c3b434bb288ffe52b5657fa 2015-10-16 13:04:01,604 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/17fc5c151c3b434bb288ffe52b5657fa, entries=466290, sequenceid=10122180, filesize=70.8 M 2015-10-16 13:04:01,605 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134286800, currentsize=23.06 MB/24180080 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 833ms, sequenceid=10122180, compaction requested=true 2015-10-16 13:04:01,605 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:04:01,606 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 6 compacting, 4 eligible, 10 blocking 2015-10-16 13:04:01,607 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio 2015-10-16 13:04:01,607 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:04:01,607 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:04:01,629 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 31026 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 973363, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:01,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 31040 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023105, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:01,853 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 31049 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1193423, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:01,962 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 31057 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:02,083 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 31074 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1074578, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:02,137 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630402_1099518555032] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 21742 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 805849, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:04:02,191 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630402_1099518555032] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 21851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 509755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:04:02,191 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:04:02,191 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:04:02,191 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:04:02,191 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:04:02,191 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:04:02,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 31081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1198063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:02,326 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 31089 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1061965, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:02,474 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 31108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1110958, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:02,550 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 31115 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1190286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:02,610 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 31119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 933081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:02,654 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 31135 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842746, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:02,741 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 31145 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1129875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:02,862 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 31153 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102550, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:02,980 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 31161 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3162027, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:03,108 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630402_1099518555032] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 27042 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489384, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:04:03,108 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:04:03,108 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:04:03,108 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:04:03,108 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:04:03,108 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:04:03,109 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 31167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1118200, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:03,207 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 31174 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:03,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 31177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1084924, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:03,325 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 31179 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984059, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:03,485 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 157ms (threshold=20ms); ack: seqno: 31185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069885, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:03,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 31187 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 886279, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:03,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 31196 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1132067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:03,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 31212 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:03,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 31229 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967691, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:03,995 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:03,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630402_1099518555032] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 32328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 448208, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:04:03,995 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:03,995 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:03,995 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:04:03,995 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:04,034 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 31245 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1109255, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:04,145 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630403_1099518555033] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 31252 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964913, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:04,270 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 31262 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1205703, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:04,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 31411 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 973705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:04,485 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 31418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:04,646 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 157ms (threshold=20ms); ack: seqno: 31425 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1251416, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:04,809 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630402_1099518555032] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 37094 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 409703, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:04:04,809 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:04,809 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:04,809 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:04,809 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:04,809 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:04,897 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 245ms (threshold=20ms); ack: seqno: 31431 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1192532, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:05,050 DEBUG [PriorityRpcServer.handler=16,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:04:05,050 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 13:04:05,050 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:04:05,050 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 6 compacting, 4 eligible, 10 blocking 2015-10-16 13:04:05,052 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio 2015-10-16 13:04:05,052 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:04:05,052 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:04:05,117 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:04:05,175 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630402_1099518555032] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 39175 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 28613503, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:04:05,175 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 13:04:05,185 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025837683 with entries=77899, filesize=122.11 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025845117 2015-10-16 13:04:05,188 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630406_1099518555036] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1103368, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:05,189 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025829639 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025829639 2015-10-16 13:04:05,240 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 340ms (threshold=20ms); ack: seqno: 31437 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1094047, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:05,367 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 31444 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1130463, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:05,503 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 31452 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1029212, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:05,656 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 31468 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845939, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:05,701 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 31486 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 773837, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:05,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 31494 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 837066, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:05,770 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630406_1099518555036] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 3199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 536387, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:05,770 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 31497 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736096, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:05,770 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:05,770 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:04:05,770 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:04:05,770 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:04:05,770 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:04:05,805 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 31503 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 855326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:05,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 31506 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813748, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:05,944 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 31520 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 730369, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:06,081 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 31528 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 918919, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:06,107 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 31535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1103719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:06,259 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 147ms (threshold=20ms); ack: seqno: 31543 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 894166, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:06,301 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 31551 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870645, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:06,328 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 31560 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1256214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:06,456 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 31569 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 793659, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:06,589 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630406_1099518555036] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 7551 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808690, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:06,590 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 31576 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 826188, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:06,694 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 31583 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1091689, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:06,761 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630406_1099518555036] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 8368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 522932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:06,792 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 13:04:06,793 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 13:04:06,793 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 13:04:06,793 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 13:04:06,793 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 70ms (threshold=20ms) 2015-10-16 13:04:06,812 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 31590 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904295, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:06,939 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 31596 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 901585, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:06,965 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 31602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 747583, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:07,076 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 31610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 785512, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:07,211 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 31616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 815714, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:07,238 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 31624 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 738790, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:07,273 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 31633 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 899745, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:07,477 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 31756 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023860, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:07,581 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 31762 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2233985, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:07,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630406_1099518555036] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 13171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 472850, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:07,689 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:07,689 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:07,689 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:07,689 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:07,694 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 31769 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 796041, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:07,809 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 31783 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 823990, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:07,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 31792 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830871, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:08,037 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 31808 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 828113, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:08,145 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 31816 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1117514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:08,250 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 31823 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835095, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:08,377 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 31842 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836791, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:08,529 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 31852 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838964, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:08,636 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630406_1099518555036] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 18302 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 415682, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:08,636 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:08,636 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:04:08,636 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:04:08,636 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:04:08,636 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:08,638 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 31873 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 978781, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:08,697 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 31884 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904982, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:08,820 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 31898 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 829960, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:08,928 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 31904 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 929124, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:09,042 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 31912 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:09,147 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 31919 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994294, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:09,254 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 31926 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964124, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:09,361 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 31932 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 727205, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:09,482 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 31939 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833663, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:09,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630406_1099518555036] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 23589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 416976, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:09,565 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:09,565 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:09,565 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:09,565 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:09,565 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:09,622 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 31956 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 746178, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:09,827 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 180ms (threshold=20ms); ack: seqno: 31974 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 858989, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:09,970 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 31981 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873486, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:10,047 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 31988 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 837910, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:10,166 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 13:04:10,166 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 29ms (threshold=20ms) 2015-10-16 13:04:10,201 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 32001 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853691, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:10,336 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 32008 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 714599, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:10,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 32014 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 733465, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:10,398 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 32023 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 697111, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:10,482 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630406_1099518555036] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 28834 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 417633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:10,482 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:04:10,482 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:04:10,482 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:04:10,482 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:04:10,482 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:04:10,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 257ms (threshold=20ms); ack: seqno: 32031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 991964, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:10,854 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 158ms (threshold=20ms); ack: seqno: 32048 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 891530, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:10,888 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 32057 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 778611, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:11,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 32195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825412, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:11,249 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 32213 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 872918, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:11,330 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630406_1099518555036] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 33618 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 442063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:11,330 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:11,330 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:11,330 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:04:11,330 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:11,330 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:11,379 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 32228 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853167, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:11,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 32235 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 897400, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:11,619 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 32249 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 850984, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:11,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 32255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853650, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:11,838 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 32262 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910643, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:11,953 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 32269 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 858712, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:12,094 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 32296 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1064689, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:12,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630406_1099518555036] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 37661 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 616223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:12,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 32304 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838154, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:12,313 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 32312 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835884, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:12,369 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 32329 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 669275, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:12,369 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630406_1099518555036] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 38845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 494034, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:12,370 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:12,370 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:04:12,370 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:12,370 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:12,370 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:04:12,441 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:04:12,466 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 32334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:12,480 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 13:04:12,493 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025845117 with entries=78033, filesize=122.33 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025852441 2015-10-16 13:04:12,684 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 32458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920845, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:12,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 32466 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1061988, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:12,920 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 32480 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 931778, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:13,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 32487 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974009, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:13,126 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 32494 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1002531, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:13,207 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630407_1099518555037] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 4154 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 394191, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:13,207 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:13,207 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:13,207 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:04:13,207 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:04:13,207 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:13,226 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 32500 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 831049, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:13,371 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 32515 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845343, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:13,510 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 32522 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963017, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:13,677 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 162ms (threshold=20ms); ack: seqno: 32531 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865364, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:13,946 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 263ms (threshold=20ms); ack: seqno: 32539 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1029987, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:14,110 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630407_1099518555037] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 9599 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 459797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:14,110 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:14,110 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:14,110 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:14,110 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:14,110 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:14,112 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 32555 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1024756, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:14,376 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 32681 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 837386, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:14,491 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 32690 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833028, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:14,621 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 32698 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911007, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:14,767 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 32723 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 806374, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:14,895 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630407_1099518555037] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 13995 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 554248, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:14,895 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 32839 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 732711, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:14,895 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:14,895 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:04:14,895 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:04:14,895 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:14,895 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:15,049 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 32874 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946428, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:15,161 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 32882 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:15,265 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 32889 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859465, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:15,422 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 32898 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:15,448 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 32905 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 976030, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:15,522 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 32932 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:15,640 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 32941 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 903913, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:15,752 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 32947 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1164929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:15,822 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:04:15,822 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630407_1099518555037] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 19116 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 482778, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:15,822 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:04:15,822 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:04:15,822 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:04:15,822 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:04:15,866 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 32956 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 840596, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:15,973 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 32964 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1042424, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:16,082 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 32970 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833076, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:16,201 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 32977 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989134, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:16,311 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 32984 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 817080, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:16,348 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 32998 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819079, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:16,440 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 33033 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 680238, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:16,566 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 33044 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 803879, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:16,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630407_1099518555037] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 24294 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 490023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:16,728 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:16,728 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:04:16,728 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:04:16,728 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:16,728 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:16,729 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 159ms (threshold=20ms); ack: seqno: 33051 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989493, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:16,863 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 33058 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887631, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:16,889 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 33064 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861419, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:16,925 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 33070 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969457, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:17,077 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 148ms (threshold=20ms); ack: seqno: 33078 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1216328, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:17,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 33085 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797054, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:17,139 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 33093 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853930, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:17,199 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 33106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 738766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:17,333 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 33110 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847906, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:17,385 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 33114 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 900980, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:17,690 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630407_1099518555037] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 29644 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489500, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:17,690 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:17,690 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:17,690 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:17,690 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:17,690 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:04:17,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 301ms (threshold=20ms); ack: seqno: 33123 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1294232, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:17,855 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 159ms (threshold=20ms); ack: seqno: 33130 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:17,907 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 33137 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 806089, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:18,036 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 33149 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:18,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 33157 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1009613, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:18,290 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 33164 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1146077, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:18,317 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 33171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:18,344 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 33179 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992976, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:18,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 33197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1024407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:18,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 33213 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1079788, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:18,757 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 134ms (threshold=20ms); ack: seqno: 33227 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1202270, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:18,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 33233 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:18,901 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 33239 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1164913, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:19,033 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 33250 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1163617, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:19,084 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 33258 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965652, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:19,222 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 33275 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1159581, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:19,336 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 33283 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 8625869, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:19,448 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 33291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 925134, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:19,560 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 33298 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089795, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:19,598 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 33301 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922684, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:19,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 134ms (threshold=20ms); ack: seqno: 33307 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1074994, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:19,761 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 33314 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 958432, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:19,788 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 33320 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905947, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:19,823 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 33328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:19,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 134ms (threshold=20ms); ack: seqno: 33332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043621, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:19,985 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630405_1099518555035] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 33339 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 880046, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:20,091 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 33355 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1719237, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:20,192 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 33372 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1504428, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:20,445 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 250ms (threshold=20ms); ack: seqno: 33379 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1629412, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:20,479 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 33386 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1402871, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:20,615 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 33392 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1455523, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:20,650 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 33399 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1363113, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:20,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 33405 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1503571, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:20,721 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 33413 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1277072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:20,772 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 33420 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:20,815 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 33428 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1226365, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:20,851 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 33437 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1064800, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:20,978 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 33445 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1328936, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:21,005 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 33453 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1302024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:21,040 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 33460 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1132041, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:21,202 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 33475 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1170730, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:21,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 33481 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1251934, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:21,281 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 33488 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1046294, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:21,315 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 33494 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014284, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:21,478 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 140ms (threshold=20ms); ack: seqno: 33510 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1361732, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:21,506 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 33519 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:21,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 33528 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1128780, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:21,793 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 148ms (threshold=20ms); ack: seqno: 33536 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1230168, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:21,819 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 33543 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915997, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:21,945 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 33550 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:21,972 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 33556 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 852085, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:22,101 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 33563 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1282558, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:22,351 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 33737 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050490, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:22,508 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 33772 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081447, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:22,627 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 33780 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1242417, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:22,812 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 33815 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1135755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:22,975 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 146ms (threshold=20ms); ack: seqno: 33830 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188053, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:23,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 33835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1123120, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:23,184 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 33844 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3167696, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:23,294 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 33863 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1103209, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:23,397 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 33870 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1135367, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:23,515 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 33883 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1039102, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:23,618 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 33890 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1034246, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:23,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 33897 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1189248, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:23,838 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 33904 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1289951, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:23,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 33911 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1327416, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:24,046 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 33920 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:24,153 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 33928 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 999932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:24,283 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 33945 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1185661, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:24,403 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 33952 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1299495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:24,504 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 33959 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1232308, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:24,701 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 196ms (threshold=20ms); ack: seqno: 33962 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1308372, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:24,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 33968 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1221626, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:24,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 33976 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986775, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:24,915 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 33983 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1228396, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:24,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 33990 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 957262, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:24,984 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 33997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920939, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:25,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 34007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 57329941, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:25,688 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 622ms (threshold=20ms); ack: seqno: 34014 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1395231, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:25,769 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 34064 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:25,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 34078 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1394124, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:26,041 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 34086 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1244506, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:26,177 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 34096 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127387, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:26,222 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 34112 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1112952, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:26,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 34119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1122913, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:26,367 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 34132 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081337, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:26,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 34138 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1349465, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:26,550 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 34149 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 860696, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:26,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630408_1099518555038] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 34168 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 957526, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:27,622 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630409_1099518555039] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 35465 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 8970321, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:04:28,528 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630409_1099518555039] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 36877 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 764596, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:04:29,470 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 38252 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 734909, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:29,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 175ms (threshold=20ms); ack: seqno: 38677 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081146, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:30,082 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 38834 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 846499, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:30,243 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 38991 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:30,422 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 39155 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1110684, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:30,515 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 39195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1087685, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:30,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 229ms (threshold=20ms); ack: seqno: 39355 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1076833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:30,919 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 39364 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1146442, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:31,015 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 39381 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1122635, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:31,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 39385 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:31,124 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 39397 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1141048, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:31,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 39416 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101106, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:31,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 39421 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1128097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:31,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 39423 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069712, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:31,421 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 39430 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1153063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:31,503 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 39442 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1152831, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:31,568 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 39449 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050778, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:31,650 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 39458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1160870, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:31,715 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 39462 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1075295, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:31,803 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 39475 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1181340, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:31,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 39480 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1100362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:31,970 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 39488 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1114048, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:32,036 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 39496 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1147944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:32,126 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=779, evicted=0, evictedPerRun=0.0 2015-10-16 13:04:32,165 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 39507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1183372, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:32,231 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 39513 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1177255, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:32,281 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 39529 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1028552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:32,352 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 39535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207280, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:32,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 39547 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1217362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:32,479 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 39551 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1147200, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:32,537 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 39561 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088882, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:32,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 39566 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196351, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:32,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630410_1099518555040] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 39584 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1228894, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:32,889 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630411_1099518555041] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 39612 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1246152, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:04:33,168 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630411_1099518555041] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 39902 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967923, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:04:33,456 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630411_1099518555041] hdfs.DFSClient: Slow ReadProcessor read fields took 185ms (threshold=20ms); ack: seqno: 40049 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1033911, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:04:33,609 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630411_1099518555041] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 40205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1108659, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:04:33,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630411_1099518555041] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 40360 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943469, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:04:33,961 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630411_1099518555041] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 40500 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1086040, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:04:34,143 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630411_1099518555041] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 40662 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:04:34,328 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#45 9630588/3436768 (280.22%), rate=33733.65 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=unlimited, activeCompactions=1] 2015-10-16 13:04:34,329 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630411_1099518555041] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 40816 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005840, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:04:34,458 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630411_1099518555041] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 40963 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794467, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:04:34,515 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630411_1099518555041] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 40965 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1166449, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:04:34,693 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630411_1099518555041] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 41123 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1016872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:04:34,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630411_1099518555041] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 41274 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1113830, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:04:35,095 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630411_1099518555041] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 41418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1222383, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:04:35,276 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630411_1099518555041] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 41570 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 973770, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:04:35,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630412_1099518555042] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 41730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 916218, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:04:35,636 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630412_1099518555042] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 41883 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1120101, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:04:35,904 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630412_1099518555042] hdfs.DFSClient: Slow ReadProcessor read fields took 182ms (threshold=20ms); ack: seqno: 41993 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1113831, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:04:36,058 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630412_1099518555042] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 42138 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050922, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:04:36,217 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630412_1099518555042] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 42312 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 712594, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:04:36,807 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630412_1099518555042] hdfs.DFSClient: Slow ReadProcessor read fields took 167ms (threshold=20ms); ack: seqno: 42949 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1094325, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:04:36,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630412_1099518555042] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 43113 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971816, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:04:37,128 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630412_1099518555042] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 43278 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966231, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:04:37,301 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630412_1099518555042] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 43426 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047086, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:04:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=752435, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 13:04:37,426 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630412_1099518555042] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 43520 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 951024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:04:37,486 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630412_1099518555042] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 43527 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174232, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:04:37,618 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630412_1099518555042] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 43674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 625980, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:04:37,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630412_1099518555042] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 43684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 978092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:04:37,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 43846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1107634, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:38,044 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 43997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1165702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:38,217 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 44158 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088101, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:38,348 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 44199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 960743, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:38,521 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 44358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005799, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:38,694 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 44508 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920786, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:38,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 44607 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825798, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 13:04:38,853 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 44611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 883754, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:39,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 44765 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154776, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:39,288 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 153ms (threshold=20ms); ack: seqno: 44851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070869, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:39,373 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 44909 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1061450, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:39,469 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 44963 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 952609, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:39,544 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 45015 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919135, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:39,586 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 45025 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 735360, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:39,654 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 45070 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1015186, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:39,732 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 45123 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956212, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:39,821 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 45170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1147558, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:39,907 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 45230 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 960045, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:40,049 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 45331 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 871554, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:40,137 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 45387 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967377, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:40,236 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 45436 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 893543, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:40,335 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 45492 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963301, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:40,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 45554 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 918590, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:40,523 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 45611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948211, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:40,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630413_1099518555043] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 45805 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1179440, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:04:40,844 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630414_1099518555044] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 45862 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1244555, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:40,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630414_1099518555044] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 45866 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1105910, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:40,989 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630414_1099518555044] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 45914 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1184321, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:41,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630414_1099518555044] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 45970 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1369707, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:41,264 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630414_1099518555044] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 46018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1108112, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:41,414 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630414_1099518555044] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 46149 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1151375, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:41,626 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630414_1099518555044] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 46395 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672154, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:42,581 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630414_1099518555044] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 47755 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 744087, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:43,502 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630415_1099518555045] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 49118 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:04:44,438 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630416_1099518555046] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 50493 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 775355, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:04:44,878 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630416_1099518555046] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 51151 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 799089, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:04:45,242 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630416_1099518555046] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 51617 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937017, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:04:45,429 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630416_1099518555046] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 51844 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 674971, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:04:46,370 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630417_1099518555047] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 53198 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 727647, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:04:47,328 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630418_1099518555048] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 54571 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:47,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630418_1099518555048] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 55399 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 748048, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:48,297 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630418_1099518555048] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 55929 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 583432, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:04:49,221 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630419_1099518555049] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 57303 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 666182, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:50,162 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630420_1099518555050] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 58663 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 708206, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:04:50,931 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630420_1099518555050] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 59730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 671084, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:04:51,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630420_1099518555050] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 60024 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 661783, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:04:52,148 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630421_1099518555051] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 61385 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 658672, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:52,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630421_1099518555051] hdfs.DFSClient: Slow ReadProcessor read fields took 195ms (threshold=20ms); ack: seqno: 62201 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:52,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630421_1099518555051] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 62255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 877226, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:53,021 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630421_1099518555051] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 62305 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870030, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:53,106 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630421_1099518555051] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 62359 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 828023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:53,178 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630421_1099518555051] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 62406 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:53,260 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630421_1099518555051] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 62463 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943301, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:04:53,432 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630422_1099518555052] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 62573 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1115267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:04:53,531 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630422_1099518555052] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 62626 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1027172, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:04:53,635 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630422_1099518555052] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 62688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 897048, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:04:53,725 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630422_1099518555052] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 62739 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 805083, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:04:53,817 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630422_1099518555052] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 62789 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834574, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:04:53,915 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630422_1099518555052] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 62842 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:04:54,013 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630422_1099518555052] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 62887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1115960, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:04:54,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630422_1099518555052] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 62949 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012207, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:04:54,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630422_1099518555052] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 63007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948025, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:04:54,312 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630422_1099518555052] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 63056 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954061, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:04:54,420 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630422_1099518555052] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 63109 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1123643, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:04:54,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630422_1099518555052] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 63162 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1031797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:04:54,652 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630422_1099518555052] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 63208 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986525, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:04:54,790 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630422_1099518555052] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 63266 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 889451, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:04:55,423 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630422_1099518555052] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 64085 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3658707, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:04:55,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630423_1099518555053] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 64698 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833178, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:56,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630423_1099518555053] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 65452 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 707978, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:04:57,307 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630424_1099518555054] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 66803 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 630599, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:04:57,630 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630424_1099518555054] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 67248 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 730986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:04:58,278 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630424_1099518555054] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 68177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 645464, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:04:58,843 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630425_1099518555055] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 69035 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 883905, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:04:59,198 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630425_1099518555055] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 69547 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 615880, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:00,144 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 70917 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 719415, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:00,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 71010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1058064, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:00,515 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 178ms (threshold=20ms); ack: seqno: 71069 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1123981, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:00,836 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 71491 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 823216, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:00,925 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 71600 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839188, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:01,107 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 71802 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 935711, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:01,265 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 152ms (threshold=20ms); ack: seqno: 71810 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1165834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:01,424 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 71958 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896162, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:01,544 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 72037 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026336, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:01,693 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 72183 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048457, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:01,754 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 72188 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:01,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 72268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 698444, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:01,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 72275 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941253, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:02,072 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 72430 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1028224, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:02,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 72584 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941330, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:02,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630426_1099518555056] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 72741 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1002984, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:03,112 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630427_1099518555057] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 73551 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 935767, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:05:03,210 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630427_1099518555057] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 73638 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 818460, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:05:04,253 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630427_1099518555057] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 74985 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 664833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:05:05,276 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630428_1099518555058] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 76339 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 650482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:06,298 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 77683 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 715031, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:06,762 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 234ms (threshold=20ms); ack: seqno: 78022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1375981, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:07,123 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 356ms (threshold=20ms); ack: seqno: 78033 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1299785, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:07,383 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 247ms (threshold=20ms); ack: seqno: 78046 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1580517, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:07,719 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 331ms (threshold=20ms); ack: seqno: 78054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1923376, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:07,796 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 78069 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1241027, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:08,527 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 360ms (threshold=20ms); ack: seqno: 78574 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1407872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:08,605 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 78588 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1344234, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:08,903 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 272ms (threshold=20ms); ack: seqno: 78629 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1353862, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:09,051 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 78742 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 894236, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:09,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 78788 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 925526, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:09,209 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 78845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920053, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:09,297 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 78894 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:09,396 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 78944 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1025204, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:09,499 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 79003 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847740, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:09,561 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 79050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 689191, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:09,593 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 79057 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 793308, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:09,688 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 79107 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 745049, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:09,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630429_1099518555059] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 79167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937417, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:09,885 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 79218 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1091783, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:10,006 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 79273 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1049366, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:10,112 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 79338 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013137, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:10,221 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 79390 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2398770, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:10,329 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 79444 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062956, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:10,444 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 79502 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 893545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:10,573 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 79550 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1117439, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:10,704 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 79598 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975381, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:10,828 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 79652 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 899648, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:10,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 79697 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:11,099 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 79755 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1052541, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:11,237 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 79764 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:11,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 79944 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 858967, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:11,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 80109 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 880111, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:11,750 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 80273 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909027, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:11,855 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 80393 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729517, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:11,921 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 80402 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936605, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:11,956 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 80404 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932399, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:12,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 80415 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844169, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:12,091 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 80469 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 724186, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:12,130 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 80482 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 740474, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:12,252 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 80540 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 879237, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:12,437 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 80710 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1015170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:12,566 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 80750 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1058770, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:12,629 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 80759 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1065297, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:12,710 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 80834 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 753598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:12,805 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 80848 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1059199, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:12,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 80866 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1001715, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:13,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 80931 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014583, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:13,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 80992 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 829499, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:13,252 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 80999 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 987702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:13,347 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 81006 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006868, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:13,468 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 81022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1160808, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:13,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 81026 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1063090, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:13,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 81040 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1095470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:13,777 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 81122 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965048, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:13,947 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630430_1099518555060] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 81271 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904094, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:05:14,117 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630431_1099518555061] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 81430 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043369, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:14,299 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630431_1099518555061] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 81587 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932618, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:14,470 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630431_1099518555061] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 81747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1042957, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:14,566 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630431_1099518555061] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 81764 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1045432, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:14,606 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630431_1099518555061] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 81778 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873077, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:14,664 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630431_1099518555061] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 81784 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 973132, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:14,768 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630431_1099518555061] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 81816 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993258, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:14,939 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630431_1099518555061] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 81962 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1125094, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:15,132 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630431_1099518555061] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 82110 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979164, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:15,307 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630431_1099518555061] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 82246 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1056609, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:15,482 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630431_1099518555061] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 82400 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020750, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:15,993 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630431_1099518555061] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 83125 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 722477, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:16,980 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630432_1099518555062] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 84480 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 733040, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:05:17,698 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630432_1099518555062] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 85399 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 962191, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:05:17,933 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 85638 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847846, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:18,029 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 85729 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7466974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:18,107 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 85810 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868464, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:18,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 85845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 666060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:18,290 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 85908 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1058993, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:18,388 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 85970 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917687, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:18,450 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 85974 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932471, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:18,538 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 86028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896226, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:18,635 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 86102 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 822328, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:18,690 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 86107 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 901996, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:18,777 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 86167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927783, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:18,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 86172 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898782, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:18,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 86225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1061939, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:18,960 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 86232 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984746, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:19,055 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 86307 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804850, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:19,115 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 86314 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934166, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:19,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 86386 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965853, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:19,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 86391 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989213, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:19,386 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 86456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942370, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:19,502 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 86467 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1172554, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:19,546 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 86475 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921311, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:19,709 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 157ms (threshold=20ms); ack: seqno: 86486 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1148313, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:19,768 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 86492 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 990604, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:19,832 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 86509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1164711, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:19,886 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 86513 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:19,945 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 86527 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922136, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:20,004 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 86532 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:20,127 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 86614 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926901, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:20,207 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 86630 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922106, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:20,347 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 86728 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950149, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:20,517 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 86894 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928320, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:20,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 87008 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797449, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:20,844 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 87170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 749591, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:20,900 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 87204 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 682228, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:20,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 87212 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 802244, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:21,127 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630433_1099518555063] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 87371 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:05:21,323 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630434_1099518555064] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 87540 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1333662, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:05:21,518 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630434_1099518555064] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 87692 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:05:21,688 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630434_1099518555064] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 87858 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1095622, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:05:22,224 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630434_1099518555064] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 88577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 766998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:05:23,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630435_1099518555065] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 89937 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736783, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:23,633 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630435_1099518555065] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 90571 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 981953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:23,860 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630435_1099518555065] hdfs.DFSClient: Slow ReadProcessor read fields took 212ms (threshold=20ms); ack: seqno: 90586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1550596, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:24,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630435_1099518555065] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 90769 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1129292, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:24,112 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630435_1099518555065] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 90814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 987107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:24,193 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#45 average throughput is 31.87 MB/sec, slept 33 time(s) and total slept time is 19995 ms. 0 active compactions remaining, total limit is unlimited 2015-10-16 13:05:24,232 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5c6694faa1b4472f963682d4511dbcf8 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5c6694faa1b4472f963682d4511dbcf8 2015-10-16 13:05:24,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630407_1099518555037] hdfs.DFSClient: Slow ReadProcessor read fields took 6367ms (threshold=20ms); ack: seqno: 30767 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 725605, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:24,245 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:05:24,246 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:05:24,254 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1f08a83c576d4583bc9945047a56a4d7, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1f08a83c576d4583bc9945047a56a4d7 2015-10-16 13:05:24,262 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b0b6dc71140942608d43144b078b55be, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b0b6dc71140942608d43144b078b55be 2015-10-16 13:05:24,271 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d7e1cbe9289b40f0919088b7a88c8df8, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d7e1cbe9289b40f0919088b7a88c8df8 2015-10-16 13:05:24,279 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/033fa9fcecaf4102a78064e2d3004d9a, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/033fa9fcecaf4102a78064e2d3004d9a 2015-10-16 13:05:24,287 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d501fca2d60342fbbe6796f418616e98, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/d501fca2d60342fbbe6796f418616e98 2015-10-16 13:05:24,296 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6e2adcb451d6484db55010b93e2befa9, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6e2adcb451d6484db55010b93e2befa9 2015-10-16 13:05:24,296 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 6 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 5c6694faa1b4472f963682d4511dbcf8(size=5.5 G), total size for store is 15.0 G. This selection was in queue for 0sec, and took 2mins, 50sec to execute. 2015-10-16 13:05:24,296 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=6, fileSize=5.5 G, priority=3, time=36903662557729038; duration=2mins, 50sec 2015-10-16 13:05:24,296 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:05:24,402 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536921216 > lower bound=16777216 2015-10-16 13:05:24,402 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 13:05:24,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630407_1099518555037] hdfs.DFSClient: Slow ReadProcessor read fields took 255ms (threshold=20ms); ack: seqno: 30768 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 742922, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:24,953 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630436_1099518555066] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 752 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 747613, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:27,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10308621, memsize=512.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/f274292ba2724b7cb5e4e1a3d18eac2f 2015-10-16 13:05:27,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/f274292ba2724b7cb5e4e1a3d18eac2f as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f274292ba2724b7cb5e4e1a3d18eac2f 2015-10-16 13:05:27,174 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f274292ba2724b7cb5e4e1a3d18eac2f, entries=1864360, sequenceid=10308621, filesize=283.2 M 2015-10-16 13:05:27,176 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630407_1099518555037] hdfs.DFSClient: Slow ReadProcessor read fields took 2676ms (threshold=20ms); ack: seqno: 30769 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 742514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:27,176 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536920800, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2774ms, sequenceid=10308621, compaction requested=true 2015-10-16 13:05:27,176 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:05:27,176 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 0 compacting, 7 eligible, 10 blocking 2015-10-16 13:05:27,180 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 5 files of size 1357404949 starting at candidate #2 after considering 15 permutations with 2 in ratio 2015-10-16 13:05:27,180 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:05:27,180 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:05:27,180 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=1.3 G 2015-10-16 13:05:27,180 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ed793e60b87549749e48c76c641adea0, keycount=3729520, bloomtype=ROW, size=566.5 M, encoding=NONE, seqNum=9829411 2015-10-16 13:05:27,180 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6f9732ddcc084bb598a1a1af24cb69e5, keycount=1898560, bloomtype=ROW, size=288.4 M, encoding=NONE, seqNum=10019280 2015-10-16 13:05:27,180 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eda6f40996a1498e9a28b55a0344caa8, keycount=562620, bloomtype=ROW, size=85.5 M, encoding=NONE, seqNum=10075547 2015-10-16 13:05:27,181 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/17fc5c151c3b434bb288ffe52b5657fa, keycount=466290, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=10122180 2015-10-16 13:05:27,181 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f274292ba2724b7cb5e4e1a3d18eac2f, keycount=1864360, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=10308621 2015-10-16 13:05:27,244 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:05:27,847 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630439_1099518555069] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 823476, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:28,774 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630440_1099518555070] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 2271 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 760495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:05:29,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630440_1099518555070] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 3640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 700796, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:05:29,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630440_1099518555070] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 3959 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862557, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:05:30,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630440_1099518555070] hdfs.DFSClient: Slow ReadProcessor read fields took 177ms (threshold=20ms); ack: seqno: 4058 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994692, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:05:30,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630441_1099518555071] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 4226 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127356, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:30,848 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630441_1099518555071] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 5004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729033, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:05:31,751 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630442_1099518555072] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 6363 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 746407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:05:31,985 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630442_1099518555072] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 6672 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 764833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:05:32,661 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630442_1099518555072] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 7725 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 711861, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:05:32,718 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630442_1099518555072] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 7772 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966859, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:05:33,562 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630443_1099518555073] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 9102 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 633099, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:05:34,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630444_1099518555074] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 10463 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 796896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:05:35,014 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630444_1099518555074] hdfs.DFSClient: Slow ReadProcessor read fields took 191ms (threshold=20ms); ack: seqno: 11004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019739, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:05:35,198 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630444_1099518555074] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 11169 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1155315, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:05:35,515 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630444_1099518555074] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 11616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 848618, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:05:35,680 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630444_1099518555074] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 11821 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1687107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:05:35,715 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630444_1099518555074] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 11825 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 732467, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:05:35,779 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630444_1099518555074] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 11835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 931029, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:05:35,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630444_1099518555074] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 11996 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1030755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:05:36,153 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630444_1099518555074] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 12156 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 867452, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:05:36,257 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630444_1099518555074] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 12207 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 901286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:05:36,451 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630444_1099518555074] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 12402 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995349, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:05:36,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630445_1099518555075] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 12641 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1200773, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:36,900 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630445_1099518555075] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 12798 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1142972, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:37,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630445_1099518555075] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 12963 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1059776, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:37,241 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630445_1099518555075] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 13121 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1068753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:37,310 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630445_1099518555075] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 13183 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692999, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:37,370 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630445_1099518555075] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 13190 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869146, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:37,543 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630445_1099518555075] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 13349 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1007449, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:38,274 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630445_1099518555075] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 14399 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966033, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:38,398 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630445_1099518555075] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 14543 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 647265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:05:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.5714285714285714, tune compaction throughput to 15.71 MB/sec 2015-10-16 13:05:39,465 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630446_1099518555076] hdfs.DFSClient: Slow ReadProcessor read fields took 534ms (threshold=20ms); ack: seqno: 15398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1387789, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:40,100 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630446_1099518555076] hdfs.DFSClient: Slow ReadProcessor read fields took 531ms (threshold=20ms); ack: seqno: 15567 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1436497, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:40,736 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630446_1099518555076] hdfs.DFSClient: Slow ReadProcessor read fields took 522ms (threshold=20ms); ack: seqno: 15734 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1387199, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:40,851 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630446_1099518555076] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 15740 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1249394, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:41,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630446_1099518555076] hdfs.DFSClient: Slow ReadProcessor read fields took 418ms (threshold=20ms); ack: seqno: 15901 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1398027, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:41,514 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630446_1099518555076] hdfs.DFSClient: Slow ReadProcessor read fields took 137ms (threshold=20ms); ack: seqno: 15908 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1572850, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:42,009 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630446_1099518555076] hdfs.DFSClient: Slow ReadProcessor read fields took 395ms (threshold=20ms); ack: seqno: 16068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1392235, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:42,068 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630446_1099518555076] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 16075 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1330722, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:42,645 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630446_1099518555076] hdfs.DFSClient: Slow ReadProcessor read fields took 478ms (threshold=20ms); ack: seqno: 16237 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1325024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:43,281 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630446_1099518555076] hdfs.DFSClient: Slow ReadProcessor read fields took 523ms (threshold=20ms); ack: seqno: 16404 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1344650, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:43,309 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630446_1099518555076] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 16411 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1040537, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:43,918 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630446_1099518555076] hdfs.DFSClient: Slow ReadProcessor read fields took 506ms (threshold=20ms); ack: seqno: 16571 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1504931, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:05:44,552 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 505ms (threshold=20ms); ack: seqno: 16740 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1215294, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:45,189 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 532ms (threshold=20ms); ack: seqno: 16907 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1572933, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:45,825 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 509ms (threshold=20ms); ack: seqno: 17074 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1455771, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:46,231 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630407_1099518555037] hdfs.DFSClient: Slow ReadProcessor read fields took 19055ms (threshold=20ms); ack: seqno: 30770 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936895, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:46,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 533ms (threshold=20ms); ack: seqno: 17241 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1226691, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:47,097 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 520ms (threshold=20ms); ack: seqno: 17408 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038851, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:47,734 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 515ms (threshold=20ms); ack: seqno: 17575 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1065876, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:48,370 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 531ms (threshold=20ms); ack: seqno: 17744 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037874, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:48,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 17752 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 958131, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:49,006 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630407_1099518555037] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 35273 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 475839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:49,006 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:05:49,008 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 478ms (threshold=20ms); ack: seqno: 17911 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101950, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:49,190 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630407_1099518555037] hdfs.DFSClient: Slow ReadProcessor read fields took 141ms (threshold=20ms); ack: seqno: 35317 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 752998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:49,645 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 533ms (threshold=20ms); ack: seqno: 18078 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1082870, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:50,281 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 530ms (threshold=20ms); ack: seqno: 18245 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1169359, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:50,361 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630407_1099518555037] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 36355 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2157950, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:50,361 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 18290 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2299595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:50,361 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:05:50,361 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:05:50,361 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:05:50,361 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:05:50,917 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 471ms (threshold=20ms); ack: seqno: 18412 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1118624, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:51,554 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 518ms (threshold=20ms); ack: seqno: 18579 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044148, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:51,613 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 18586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1519506, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:51,769 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 18741 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 822439, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:52,190 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 417ms (threshold=20ms); ack: seqno: 18746 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1095529, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:52,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630447_1099518555077] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 18750 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1040494, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:05:52,306 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 18824 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3049734, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:52,306 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630407_1099518555037] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 38217 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 387419, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:52,306 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:52,370 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 18840 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1255716, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:52,827 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 407ms (threshold=20ms); ack: seqno: 18915 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2220103, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:52,898 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 18925 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1079755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:53,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 427ms (threshold=20ms); ack: seqno: 19084 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1235388, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:53,588 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:05:53,588 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630407_1099518555037] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 42512 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 513914, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:05:53,588 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 19213 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 852981, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:53,588 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:05:53,588 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:05:53,588 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:05:53,588 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:05:53,693 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:05:53,728 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025852441 with entries=77878, filesize=122.08 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025953693 2015-10-16 13:05:53,729 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025837683 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025837683 2015-10-16 13:05:53,736 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025845117 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025845117 2015-10-16 13:05:54,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 483ms (threshold=20ms); ack: seqno: 19251 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1453140, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:54,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630449_1099518555079] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 3820 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 656674, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:05:54,418 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:05:54,418 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:05:54,418 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:05:54,418 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:05:54,418 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:05:54,734 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 509ms (threshold=20ms); ack: seqno: 19418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1353695, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:54,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 19563 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 886420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:55,268 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630449_1099518555079] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 8738 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 712755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:05:55,268 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:05:55,268 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:05:55,268 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:05:55,268 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:05:55,268 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:05:55,370 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 476ms (threshold=20ms); ack: seqno: 19585 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1354265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:56,006 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 506ms (threshold=20ms); ack: seqno: 19752 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1313188, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:56,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630449_1099518555079] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 13395 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 337761, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:05:56,079 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:05:56,079 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:05:56,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 19790 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2613389, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:56,079 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:05:56,079 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:05:56,079 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:05:56,310 DEBUG [PriorityRpcServer.handler=16,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:05:56,310 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134276160 > lower bound=16777216 2015-10-16 13:05:56,310 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:05:56,333 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630449_1099518555079] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 14808 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 600172, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:05:56,642 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 456ms (threshold=20ms); ack: seqno: 19919 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1118612, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:56,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 20003 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 774047, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:56,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630449_1099518555079] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 16890 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 510539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:05:56,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630450_1099518555080] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 548 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 756474, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:05:56,745 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:56,745 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:56,745 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:56,745 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:56,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 20011 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 722664, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:57,135 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10355259, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a433f5d43ec144e88fb32f0dfed06768 2015-10-16 13:05:57,140 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/a433f5d43ec144e88fb32f0dfed06768 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a433f5d43ec144e88fb32f0dfed06768 2015-10-16 13:05:57,147 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a433f5d43ec144e88fb32f0dfed06768, entries=466340, sequenceid=10355259, filesize=70.8 M 2015-10-16 13:05:57,148 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134300800, currentsize=24.38 MB/25559200 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 838ms, sequenceid=10355259, compaction requested=true 2015-10-16 13:05:57,148 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:05:57,148 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 5 compacting, 1 eligible, 10 blocking 2015-10-16 13:05:57,149 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:05:57,149 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:05:57,149 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:05:57,278 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 385ms (threshold=20ms); ack: seqno: 20088 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1413521, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:57,342 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 20096 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1222911, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:57,418 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 20147 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 659088, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:57,418 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630449_1099518555079] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 20634 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 393839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:05:57,418 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:57,418 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:57,418 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:57,419 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:05:57,479 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 20153 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1077790, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:57,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 358ms (threshold=20ms); ack: seqno: 20255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1237277, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:58,209 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630449_1099518555079] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 25028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 603805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:05:58,209 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:58,209 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:58,210 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:58,210 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:58,210 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:58,551 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 519ms (threshold=20ms); ack: seqno: 20422 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1372210, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:58,755 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630449_1099518555079] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 27461 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857989, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:05:59,187 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 512ms (threshold=20ms); ack: seqno: 20589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1487669, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:59,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630449_1099518555079] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 29895 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 495135, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:05:59,228 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:05:59,228 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:59,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 20592 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1184706, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:59,228 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:05:59,228 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:59,823 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630448_1099518555078] hdfs.DFSClient: Slow ReadProcessor read fields took 471ms (threshold=20ms); ack: seqno: 20756 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1341906, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:05:59,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630449_1099518555079] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 34116 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 482988, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:05:59,995 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:59,995 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:59,995 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:05:59,995 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:05:59,995 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:00,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630451_1099518555081] hdfs.DFSClient: Slow ReadProcessor read fields took 508ms (threshold=20ms); ack: seqno: 20926 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207340, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:00,567 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#49 average throughput is 37.78 MB/sec, slept 34 time(s) and total slept time is 16487 ms. 0 active compactions remaining, total limit is 15.71 MB/sec 2015-10-16 13:06:00,585 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/bd400d9a06d340e8a2bbd497821ef103 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bd400d9a06d340e8a2bbd497821ef103 2015-10-16 13:06:00,599 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:06:00,603 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:06:00,610 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ed793e60b87549749e48c76c641adea0, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ed793e60b87549749e48c76c641adea0 2015-10-16 13:06:00,618 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6f9732ddcc084bb598a1a1af24cb69e5, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/6f9732ddcc084bb598a1a1af24cb69e5 2015-10-16 13:06:00,627 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eda6f40996a1498e9a28b55a0344caa8, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eda6f40996a1498e9a28b55a0344caa8 2015-10-16 13:06:00,635 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/17fc5c151c3b434bb288ffe52b5657fa, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/17fc5c151c3b434bb288ffe52b5657fa 2015-10-16 13:06:00,643 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f274292ba2724b7cb5e4e1a3d18eac2f, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f274292ba2724b7cb5e4e1a3d18eac2f 2015-10-16 13:06:00,644 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into bd400d9a06d340e8a2bbd497821ef103(size=1.3 G), total size for store is 15.4 G. This selection was in queue for 0sec, and took 33sec to execute. 2015-10-16 13:06:00,644 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=5, fileSize=1.3 G, priority=3, time=36903836034081060; duration=33sec 2015-10-16 13:06:00,644 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:06:00,687 DEBUG [PriorityRpcServer.handler=5,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:06:00,687 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134281216 > lower bound=16777216 2015-10-16 13:06:00,687 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 13:06:00,716 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630449_1099518555079] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 38250 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 479665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:00,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630452_1099518555082] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 118 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2252138, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:00,858 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:00,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630449_1099518555079] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 38801 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 427564, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:00,859 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:00,859 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:00,859 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:00,859 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:00,900 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:06:00,944 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025953693 with entries=77864, filesize=122.06 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025960900 2015-10-16 13:06:00,946 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025852441 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025852441 2015-10-16 13:06:00,948 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630453_1099518555083] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1087997, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:01,545 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630452_1099518555082] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 1103 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 680152, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:01,545 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630453_1099518555083] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 3234 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 434362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:01,545 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:01,545 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:01,545 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:01,545 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:01,546 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:01,585 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10401895, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/802fcd66ae874d519fc051f0de19a04c 2015-10-16 13:06:01,591 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/802fcd66ae874d519fc051f0de19a04c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/802fcd66ae874d519fc051f0de19a04c 2015-10-16 13:06:01,597 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/802fcd66ae874d519fc051f0de19a04c, entries=466310, sequenceid=10401895, filesize=70.8 M 2015-10-16 13:06:01,598 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134293760, currentsize=24.57 MB/25760560 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 911ms, sequenceid=10401895, compaction requested=true 2015-10-16 13:06:01,598 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:06:01,599 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 5 store files, 0 compacting, 5 eligible, 10 blocking 2015-10-16 13:06:01,600 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 6 permutations with 0 in ratio 2015-10-16 13:06:01,601 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:06:01,601 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:06:02,420 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630453_1099518555083] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 8590 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 362586, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:02,420 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:02,420 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:02,420 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:02,420 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:03,342 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:03,342 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:03,342 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:03,342 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:03,343 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630453_1099518555083] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 14138 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 622183, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:04,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630453_1099518555083] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 19711 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 474813, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:04,319 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:04,319 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:04,319 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:04,319 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:04,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630453_1099518555083] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 21211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827679, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:04,945 DEBUG [PriorityRpcServer.handler=19,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:06:04,946 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134272816 > lower bound=16777216 2015-10-16 13:06:04,946 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 13:06:04,970 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630453_1099518555083] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 22599 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 538243, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:05,323 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630453_1099518555083] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 24311 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672270, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:05,323 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630454_1099518555084] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 486 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 759649, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:05,323 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:05,323 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:05,323 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:05,323 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:05,323 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:05,752 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10448523, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b561e3d4f72b41e982289353f4af41a4 2015-10-16 13:06:05,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/b561e3d4f72b41e982289353f4af41a4 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b561e3d4f72b41e982289353f4af41a4 2015-10-16 13:06:05,763 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b561e3d4f72b41e982289353f4af41a4, entries=466240, sequenceid=10448523, filesize=70.8 M 2015-10-16 13:06:05,765 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134272400, currentsize=23.01 MB/24127760 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 818ms, sequenceid=10448523, compaction requested=true 2015-10-16 13:06:05,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:06:05,765 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 6 store files, 0 compacting, 6 eligible, 10 blocking 2015-10-16 13:06:05,767 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222839678 starting at candidate #3 after considering 10 permutations with 1 in ratio 2015-10-16 13:06:05,767 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:06:05,767 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:06:05,768 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 13:06:05,768 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a433f5d43ec144e88fb32f0dfed06768, keycount=466340, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=10355259 2015-10-16 13:06:05,768 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/802fcd66ae874d519fc051f0de19a04c, keycount=466310, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=10401895 2015-10-16 13:06:05,768 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b561e3d4f72b41e982289353f4af41a4, keycount=466240, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=10448523 2015-10-16 13:06:05,784 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:06:05,988 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630453_1099518555083] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 27823 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 484349, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:05,988 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:05,988 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:05,988 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:05,988 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:05,988 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:06,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630455_1099518555085] hdfs.DFSClient: Slow ReadProcessor read fields took 503ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963745, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:06,826 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630453_1099518555083] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 32718 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 447556, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:06,826 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:06,826 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:06,827 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:06,826 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:06,826 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:07,067 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630455_1099518555085] hdfs.DFSClient: Slow ReadProcessor read fields took 507ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1152491, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:07,315 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630453_1099518555083] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 35500 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26392018, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:07,315 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 13:06:07,316 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 13:06:07,316 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 27ms (threshold=20ms) 2015-10-16 13:06:07,316 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 13:06:07,316 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 13:06:07,705 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630453_1099518555083] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 37578 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 525219, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:07,705 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:07,705 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:07,705 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:07,705 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:07,705 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:07,706 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630455_1099518555085] hdfs.DFSClient: Slow ReadProcessor read fields took 508ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1092000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:07,964 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:06:07,993 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025960900 with entries=77808, filesize=121.97 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025967965 2015-10-16 13:06:07,995 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025953693 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025953693 2015-10-16 13:06:08,343 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630455_1099518555085] hdfs.DFSClient: Slow ReadProcessor read fields took 515ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111519, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:08,491 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630455_1099518555085] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 804 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 730597, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:08,491 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630456_1099518555086] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 2586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 557032, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:08,491 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:08,491 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:08,491 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:08,491 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:08,491 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:08,980 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630455_1099518555085] hdfs.DFSClient: Slow ReadProcessor read fields took 467ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1082583, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:09,214 DEBUG [PriorityRpcServer.handler=1,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:06:09,214 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134273872 > lower bound=16777216 2015-10-16 13:06:09,214 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 13:06:09,242 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630456_1099518555086] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 6812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 679460, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:09,348 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630456_1099518555086] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 7128 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 523391, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:09,348 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630457_1099518555087] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 76 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946863, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:09,348 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:06:09,348 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:06:09,348 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:06:09,348 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:06:09,348 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:06:09,615 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630455_1099518555085] hdfs.DFSClient: Slow ReadProcessor read fields took 511ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1256244, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:09,990 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630456_1099518555086] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 10344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 568076, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:09,990 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630457_1099518555087] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 945 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 717780, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:09,990 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:09,990 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:09,990 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:09,990 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:09,990 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:10,168 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10495159, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/e98cdcf3ec40472e9cb6b35967503539 2015-10-16 13:06:10,174 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/e98cdcf3ec40472e9cb6b35967503539 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e98cdcf3ec40472e9cb6b35967503539 2015-10-16 13:06:10,196 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e98cdcf3ec40472e9cb6b35967503539, entries=466320, sequenceid=10495159, filesize=70.8 M 2015-10-16 13:06:10,198 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134295920, currentsize=25.94 MB/27197840 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 984ms, sequenceid=10495159, compaction requested=true 2015-10-16 13:06:10,198 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:06:10,198 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 3 compacting, 1 eligible, 10 blocking 2015-10-16 13:06:10,199 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:06:10,199 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:06:10,199 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:06:10,251 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630455_1099518555085] hdfs.DFSClient: Slow ReadProcessor read fields took 512ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1259834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:10,649 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630456_1099518555086] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 13291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 944154, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:10,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630455_1099518555085] hdfs.DFSClient: Slow ReadProcessor read fields took 516ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1383165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:10,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630455_1099518555085] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 1348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1158155, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:10,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630456_1099518555086] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 14701 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 410868, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:10,938 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:10,938 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:10,938 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:10,938 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:10,938 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:11,523 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630455_1099518555085] hdfs.DFSClient: Slow ReadProcessor read fields took 468ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1352218, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:11,691 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630456_1099518555086] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 18942 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 435930, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:11,691 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630455_1099518555085] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 1665 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 721643, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:11,691 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:11,691 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:11,691 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:11,691 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:11,691 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:12,160 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630455_1099518555085] hdfs.DFSClient: Slow ReadProcessor read fields took 466ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1247894, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:12,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630456_1099518555086] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 23754 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 536314, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:12,583 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:12,583 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:12,583 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:12,583 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:12,583 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:12,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630455_1099518555085] hdfs.DFSClient: Slow ReadProcessor read fields took 518ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1181283, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:13,204 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630456_1099518555086] hdfs.DFSClient: Slow ReadProcessor read fields took 225ms (threshold=20ms); ack: seqno: 26144 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 224227063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:13,204 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 225ms (threshold=20ms) 2015-10-16 13:06:13,204 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 225ms (threshold=20ms) 2015-10-16 13:06:13,204 INFO [sync.0] wal.FSHLog: Slow sync cost: 224 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:13,204 INFO [sync.2] wal.FSHLog: Slow sync cost: 225 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:13,204 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 225ms (threshold=20ms) 2015-10-16 13:06:13,204 INFO [sync.3] wal.FSHLog: Slow sync cost: 225 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:13,204 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 225ms (threshold=20ms) 2015-10-16 13:06:13,205 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 225ms (threshold=20ms) 2015-10-16 13:06:13,205 INFO [sync.4] wal.FSHLog: Slow sync cost: 225 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:13,205 INFO [sync.1] wal.FSHLog: Slow sync cost: 225 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:13,416 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630456_1099518555086] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 27124 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 25532265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:13,416 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 13:06:13,416 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 26ms (threshold=20ms) 2015-10-16 13:06:13,416 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 13:06:13,416 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 13:06:13,416 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 13:06:13,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630455_1099518555085] hdfs.DFSClient: Slow ReadProcessor read fields took 531ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 15836874, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:13,588 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630456_1099518555086] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 27926 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 447414, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:13,588 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630458_1099518555088] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 2170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 607602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:06:13,589 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:13,589 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:13,589 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:13,589 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:13,589 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:13,958 DEBUG [PriorityRpcServer.handler=11,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:06:13,958 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134285952 > lower bound=16777216 2015-10-16 13:06:13,958 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 13:06:14,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630458_1099518555088] hdfs.DFSClient: Slow ReadProcessor read fields took 478ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1240713, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:06:14,328 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630459_1099518555089] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 416 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 776177, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:14,328 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630456_1099518555086] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 31995 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 422072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:14,328 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:14,328 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:14,328 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:14,328 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:14,328 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:14,703 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630458_1099518555088] hdfs.DFSClient: Slow ReadProcessor read fields took 517ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1133929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:06:14,826 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10541796, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/323aeea710f9476f835fee40c92e301d 2015-10-16 13:06:14,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/323aeea710f9476f835fee40c92e301d as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/323aeea710f9476f835fee40c92e301d 2015-10-16 13:06:14,837 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/323aeea710f9476f835fee40c92e301d, entries=466330, sequenceid=10541796, filesize=70.8 M 2015-10-16 13:06:14,840 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134297920, currentsize=24.26 MB/25440800 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 881ms, sequenceid=10541796, compaction requested=true 2015-10-16 13:06:14,840 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:06:14,840 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 3 compacting, 2 eligible, 10 blocking 2015-10-16 13:06:14,841 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:06:14,841 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:06:14,841 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:06:14,989 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630456_1099518555086] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 35397 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 437741, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:14,989 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:14,989 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:14,989 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:14,989 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:14,989 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:15,339 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630458_1099518555088] hdfs.DFSClient: Slow ReadProcessor read fields took 520ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1055093, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:06:15,598 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:06:15,635 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025967965 with entries=77921, filesize=122.15 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025975598 2015-10-16 13:06:15,636 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025960900 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025960900 2015-10-16 13:06:15,841 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630460_1099518555090] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 1081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 488000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:06:15,841 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:15,841 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:15,841 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:15,841 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:15,841 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:15,975 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630458_1099518555088] hdfs.DFSClient: Slow ReadProcessor read fields took 521ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1150578, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:06:16,494 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630460_1099518555090] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 4175 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 818762, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:06:16,611 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630458_1099518555088] hdfs.DFSClient: Slow ReadProcessor read fields took 504ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1119467, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:06:16,749 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630460_1099518555090] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 5393 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 657609, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:06:16,749 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630458_1099518555088] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 2966 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 637587, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:06:16,749 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:16,749 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:16,749 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:16,749 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:16,749 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:17,247 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630458_1099518555088] hdfs.DFSClient: Slow ReadProcessor read fields took 461ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1083442, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:06:17,584 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630460_1099518555090] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 10082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 365097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:06:17,584 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:17,584 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:17,584 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:17,584 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:17,584 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:17,883 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630458_1099518555088] hdfs.DFSClient: Slow ReadProcessor read fields took 497ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1296433, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:06:18,339 DEBUG [PriorityRpcServer.handler=17,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:06:18,339 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134254880 > lower bound=16777216 2015-10-16 13:06:18,354 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.44 MB 2015-10-16 13:06:18,383 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630460_1099518555090] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 14747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 572854, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:06:18,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630460_1099518555090] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 14889 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 561023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:06:18,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630461_1099518555091] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 2 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2490445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:18,447 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:18,447 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:18,447 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:18,447 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:18,447 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:18,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630458_1099518555088] hdfs.DFSClient: Slow ReadProcessor read fields took 500ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1153681, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:06:18,617 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#50 average throughput is 16.11 MB/sec, slept 20 time(s) and total slept time is 9971 ms. 0 active compactions remaining, total limit is 15.71 MB/sec 2015-10-16 13:06:18,631 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/7ab59b8a99514240b3cc1f59adfe9fed as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7ab59b8a99514240b3cc1f59adfe9fed 2015-10-16 13:06:18,638 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:06:18,642 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:06:18,651 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a433f5d43ec144e88fb32f0dfed06768, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/a433f5d43ec144e88fb32f0dfed06768 2015-10-16 13:06:18,659 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/802fcd66ae874d519fc051f0de19a04c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/802fcd66ae874d519fc051f0de19a04c 2015-10-16 13:06:18,667 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b561e3d4f72b41e982289353f4af41a4, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/b561e3d4f72b41e982289353f4af41a4 2015-10-16 13:06:18,668 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 7ab59b8a99514240b3cc1f59adfe9fed(size=212.5 M), total size for store is 15.6 G. This selection was in queue for 0sec, and took 12sec to execute. 2015-10-16 13:06:18,668 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=4, time=36903874621435848; duration=12sec 2015-10-16 13:06:18,668 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:06:19,058 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630460_1099518555090] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 18177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 570126, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:06:19,058 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630461_1099518555091] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 946 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 774499, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:06:19,058 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:19,058 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:19,058 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:19,058 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:19,058 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:19,192 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10588586, memsize=128.5 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/192bffb1bf7b43d2ba75774f80e0c3d0 2015-10-16 13:06:19,198 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/192bffb1bf7b43d2ba75774f80e0c3d0 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/192bffb1bf7b43d2ba75774f80e0c3d0 2015-10-16 13:06:19,204 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/192bffb1bf7b43d2ba75774f80e0c3d0, entries=467860, sequenceid=10588586, filesize=71.1 M 2015-10-16 13:06:19,206 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.50 MB/134738800, currentsize=23.40 MB/24533920 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 852ms, sequenceid=10588586, compaction requested=true 2015-10-16 13:06:19,206 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:06:19,206 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 0 compacting, 7 eligible, 10 blocking 2015-10-16 13:06:19,209 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 445938239 starting at candidate #3 after considering 15 permutations with 2 in ratio 2015-10-16 13:06:19,209 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:06:19,209 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:06:19,209 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=425.3 M 2015-10-16 13:06:19,209 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7ab59b8a99514240b3cc1f59adfe9fed, keycount=1398890, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=10448523 2015-10-16 13:06:19,209 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e98cdcf3ec40472e9cb6b35967503539, keycount=466320, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=10495159 2015-10-16 13:06:19,210 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/323aeea710f9476f835fee40c92e301d, keycount=466330, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=10541796 2015-10-16 13:06:19,210 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/192bffb1bf7b43d2ba75774f80e0c3d0, keycount=467860, bloomtype=ROW, size=71.1 M, encoding=NONE, seqNum=10588586 2015-10-16 13:06:19,229 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:06:19,861 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630460_1099518555090] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 22559 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 470035, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:06:19,862 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:19,862 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:19,862 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:19,862 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:19,862 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:19,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630462_1099518555092] hdfs.DFSClient: Slow ReadProcessor read fields took 514ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1603374, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:20,508 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630462_1099518555092] hdfs.DFSClient: Slow ReadProcessor read fields took 519ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1349229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:20,649 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630460_1099518555090] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 26880 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 469619, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:06:20,649 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630462_1099518555092] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 715309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:20,649 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:06:20,649 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:06:20,649 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:06:20,649 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:06:20,649 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:06:21,144 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630462_1099518555092] hdfs.DFSClient: Slow ReadProcessor read fields took 470ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1230976, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:21,473 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630460_1099518555090] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 31618 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 409219, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:06:21,473 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:21,473 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:21,473 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:21,473 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:21,473 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:21,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630462_1099518555092] hdfs.DFSClient: Slow ReadProcessor read fields took 513ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1156556, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:21,798 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630460_1099518555090] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 32826 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741456, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:06:22,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630460_1099518555090] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 36472 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 391386, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:06:22,430 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:22,430 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:22,430 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:22,430 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:22,430 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:22,432 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630462_1099518555092] hdfs.DFSClient: Slow ReadProcessor read fields took 528ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1333251, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:22,723 DEBUG [PriorityRpcServer.handler=14,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:06:22,723 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134278560 > lower bound=16777216 2015-10-16 13:06:22,723 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.03 MB 2015-10-16 13:06:22,748 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630460_1099518555090] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 38131 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 571544, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:06:22,899 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:06:22,943 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025975598 with entries=77958, filesize=122.21 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025982899 2015-10-16 13:06:22,944 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025967965 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025967965 2015-10-16 13:06:23,067 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630462_1099518555092] hdfs.DFSClient: Slow ReadProcessor read fields took 523ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1180440, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:23,136 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630462_1099518555092] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 1040 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 648031, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:23,136 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630464_1099518555094] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 910 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 427559, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:23,136 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:23,136 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:23,136 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:23,136 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:23,136 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:23,137 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630463_1099518555093] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 466 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801135, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:06:23,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10635219, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/3939b430c93a4aaebfc46b09e2185db6 2015-10-16 13:06:23,628 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/3939b430c93a4aaebfc46b09e2185db6 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3939b430c93a4aaebfc46b09e2185db6 2015-10-16 13:06:23,636 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3939b430c93a4aaebfc46b09e2185db6, entries=466280, sequenceid=10635219, filesize=70.8 M 2015-10-16 13:06:23,638 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134284480, currentsize=26.92 MB/28223280 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 915ms, sequenceid=10635219, compaction requested=true 2015-10-16 13:06:23,638 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:06:23,638 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 4 compacting, 1 eligible, 10 blocking 2015-10-16 13:06:23,639 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:06:23,639 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:06:23,639 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:06:23,785 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630464_1099518555094] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 4429 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 557819, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:23,785 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:23,785 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:23,786 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:23,786 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:23,786 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:24,418 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630462_1099518555092] hdfs.DFSClient: Slow ReadProcessor read fields took 1186ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 714546805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:24,560 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630462_1099518555092] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 1401 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 719276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:24,560 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:24,560 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630464_1099518555094] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 9250 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 415327, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:24,560 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:24,560 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:24,560 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:24,560 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:25,118 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630462_1099518555092] hdfs.DFSClient: Slow ReadProcessor read fields took 485ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101510, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:25,334 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:25,334 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630464_1099518555094] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 13751 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 436631, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:25,334 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:25,334 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:25,334 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:25,334 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:25,752 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630462_1099518555092] hdfs.DFSClient: Slow ReadProcessor read fields took 520ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:26,151 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630464_1099518555094] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 18639 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 617213, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:26,151 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:26,151 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:26,151 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:26,151 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:26,151 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:26,390 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630462_1099518555092] hdfs.DFSClient: Slow ReadProcessor read fields took 525ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1158286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:26,664 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630464_1099518555094] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 20983 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 720340, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:26,889 DEBUG [PriorityRpcServer.handler=11,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:06:26,889 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134276496 > lower bound=16777216 2015-10-16 13:06:26,889 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 13:06:26,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630464_1099518555094] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 22362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 568023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:27,026 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630462_1099518555092] hdfs.DFSClient: Slow ReadProcessor read fields took 523ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1099128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:27,076 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630462_1099518555092] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 2013 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 949811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:27,076 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630464_1099518555094] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 23045 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 463209, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:27,076 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630465_1099518555095] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 168 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 660249, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:06:27,076 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:06:27,076 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:06:27,076 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:06:27,076 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:06:27,076 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:06:27,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630464_1099518555094] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 26336 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 364936, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:27,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630465_1099518555095] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 1055 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 631237, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:06:27,673 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:27,673 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:27,673 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:27,673 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:27,673 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:27,675 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 487ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1238002, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:27,742 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10681848, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5c0d7c395cea4f3fa017b0f71add0b9c 2015-10-16 13:06:27,747 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5c0d7c395cea4f3fa017b0f71add0b9c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5c0d7c395cea4f3fa017b0f71add0b9c 2015-10-16 13:06:27,753 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5c0d7c395cea4f3fa017b0f71add0b9c, entries=466250, sequenceid=10681848, filesize=70.8 M 2015-10-16 13:06:27,755 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134276080, currentsize=24.06 MB/25233760 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 866ms, sequenceid=10681848, compaction requested=true 2015-10-16 13:06:27,755 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:06:27,755 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 4 compacting, 2 eligible, 10 blocking 2015-10-16 13:06:27,756 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:06:27,756 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:06:27,756 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:06:28,310 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 517ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1391985, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:28,409 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630464_1099518555094] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 30507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692414, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:28,409 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 2437 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 744130, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:28,409 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:28,409 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:28,409 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:28,409 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:28,410 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:28,947 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 491ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1126704, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:29,209 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:29,209 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:29,209 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:29,209 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:29,209 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:29,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 520ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1175845, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:29,807 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:06:29,851 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025982899 with entries=77954, filesize=122.20 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025989807 2015-10-16 13:06:29,852 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025975598 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025975598 2015-10-16 13:06:29,853 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630467_1099518555097] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 826871, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:30,021 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630467_1099518555097] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 357532, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:30,021 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:30,021 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:30,021 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:30,021 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:30,022 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:30,218 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 520ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101940, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:30,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630467_1099518555097] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 5616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 486318, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:30,872 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:30,872 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:30,873 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:30,873 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:30,873 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:30,874 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 535ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054212, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:31,052 DEBUG [PriorityRpcServer.handler=17,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:06:31,052 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134279360 > lower bound=16777216 2015-10-16 13:06:31,052 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 13:06:31,435 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630467_1099518555097] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 7928 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 742167, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:31,510 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 513ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1244480, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:31,611 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630467_1099518555097] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 8695 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 407199, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:31,611 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630468_1099518555098] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 706 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 748305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:31,611 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 3268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 783633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:31,611 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:31,611 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:31,611 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:31,611 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:31,612 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:31,983 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10728480, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/62ed602fa53b442b85fa521c619fff0b 2015-10-16 13:06:31,989 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/62ed602fa53b442b85fa521c619fff0b as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/62ed602fa53b442b85fa521c619fff0b 2015-10-16 13:06:31,994 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/62ed602fa53b442b85fa521c619fff0b, entries=466280, sequenceid=10728480, filesize=70.8 M 2015-10-16 13:06:31,996 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134283840, currentsize=23.18 MB/24309280 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 944ms, sequenceid=10728480, compaction requested=true 2015-10-16 13:06:31,996 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:06:31,996 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 4 compacting, 3 eligible, 10 blocking 2015-10-16 13:06:31,998 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222826620 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 13:06:31,998 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:06:31,998 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:06:31,998 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 13:06:31,998 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3939b430c93a4aaebfc46b09e2185db6, keycount=466280, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=10635219 2015-10-16 13:06:31,998 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5c0d7c395cea4f3fa017b0f71add0b9c, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=10681848 2015-10-16 13:06:31,998 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/62ed602fa53b442b85fa521c619fff0b, keycount=466280, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=10728480 2015-10-16 13:06:32,013 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:06:32,146 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 471ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1183383, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:32,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630467_1099518555097] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 12037 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 540710, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:32,248 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:32,249 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:32,248 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:32,249 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 3423 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 731701, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:33,166 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630467_1099518555097] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 17216 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 535096, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:33,167 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:33,167 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:33,167 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:33,167 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:33,167 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:33,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630469_1099518555099] hdfs.DFSClient: Slow ReadProcessor read fields took 1151ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1134821, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:06:33,418 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 1107ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1097602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:33,911 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630467_1099518555097] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 21420 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 494247, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:33,911 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:34,562 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630469_1099518555099] hdfs.DFSClient: Slow ReadProcessor read fields took 1157ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102575, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:06:34,691 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 1152ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070337, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:34,748 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630467_1099518555097] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 26245 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 609715, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:34,748 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 3703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 634745, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:34,748 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:34,748 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:34,748 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:34,748 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:34,748 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:34,854 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630467_1099518555097] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 26701 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 9667043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:34,854 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 13:06:34,854 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 13:06:34,854 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 13:06:34,854 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 13:06:34,854 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 13:06:35,419 DEBUG [PriorityRpcServer.handler=1,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:06:35,419 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134265056 > lower bound=16777216 2015-10-16 13:06:35,419 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 13:06:35,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630467_1099518555097] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 30090 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 722398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:35,628 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630470_1099518555100] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 184 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 731424, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:35,628 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630467_1099518555097] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 30827 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489912, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:35,628 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:35,628 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:35,628 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:35,628 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:35,628 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:35,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630469_1099518555099] hdfs.DFSClient: Slow ReadProcessor read fields took 1152ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1063542, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:06:35,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 1114ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:36,147 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630467_1099518555097] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 32951 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 509282, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:36,238 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630470_1099518555100] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 1125 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767703, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:06:36,238 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630467_1099518555097] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 33273 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 476328, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:36,238 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:36,238 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:36,238 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:36,238 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:36,238 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:36,258 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10775113, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/3868aaf7bd7549fc9b596ac7877200c1 2015-10-16 13:06:36,263 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/3868aaf7bd7549fc9b596ac7877200c1 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3868aaf7bd7549fc9b596ac7877200c1 2015-10-16 13:06:36,269 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3868aaf7bd7549fc9b596ac7877200c1, entries=466290, sequenceid=10775113, filesize=70.8 M 2015-10-16 13:06:36,271 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134287680, currentsize=18.55 MB/19448080 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 852ms, sequenceid=10775113, compaction requested=true 2015-10-16 13:06:36,271 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:06:37,107 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630469_1099518555099] hdfs.DFSClient: Slow ReadProcessor read fields took 1145ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1167635, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:06:37,156 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630469_1099518555099] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 680 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 906611, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:06:37,156 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:37,156 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:37,156 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:37,156 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:37,156 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:06:37,216 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:06:37,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630466_1099518555096] hdfs.DFSClient: Slow ReadProcessor read fields took 1152ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1294388, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:06:37,267 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025989807 with entries=77932, filesize=122.16 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025997216 2015-10-16 13:06:37,269 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025982899 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025982899 2015-10-16 13:06:37,270 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932368, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:37,931 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:37,931 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:37,931 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:37,931 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 3754 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 432825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:37,932 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:37,932 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:38,384 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630469_1099518555099] hdfs.DFSClient: Slow ReadProcessor read fields took 1104ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5891440, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:06:38,507 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630472_1099518555102] hdfs.DFSClient: Slow ReadProcessor read fields took 1140ms (threshold=20ms); ack: seqno: 4182 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1264790, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:06:38,679 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:38,679 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:38,679 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 7959 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 466872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:38,679 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:38,679 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 13:06:38,894 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 9129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 23450875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:38,894 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 13:06:38,894 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 24ms (threshold=20ms) 2015-10-16 13:06:38,895 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 13:06:38,895 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 13:06:38,895 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 13:06:39,651 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630469_1099518555099] hdfs.DFSClient: Slow ReadProcessor read fields took 1147ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1218606, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:06:39,715 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 13:06:39,715 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 13:06:39,715 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630469_1099518555099] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 1004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037423, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:06:39,715 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 63ms (threshold=20ms) 2015-10-16 13:06:39,715 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 62ms (threshold=20ms) 2015-10-16 13:06:39,715 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 13562 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 498634, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:39,715 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 62ms (threshold=20ms) 2015-10-16 13:06:39,779 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630472_1099518555102] hdfs.DFSClient: Slow ReadProcessor read fields took 1158ms (threshold=20ms); ack: seqno: 4348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1314105, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:06:39,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 14173 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 18334726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:39,879 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:06:39,879 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:06:39,879 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:06:39,879 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:06:39,879 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:06:39,921 DEBUG [PriorityRpcServer.handler=6,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:06:39,922 WARN [MemStoreFlusher.0] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 13:06:39,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 13:06:40,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630472_1099518555102] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 4752 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1278277, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:06:40,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 15306 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 416667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:40,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630469_1099518555099] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 1535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779442, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:06:40,133 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:40,133 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:40,133 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:40,133 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:40,133 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:40,547 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 17388 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 614036, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:40,547 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630472_1099518555102] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 5069 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819587, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:06:40,547 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:06:40,547 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630469_1099518555099] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 2070 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 706432, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:06:40,547 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:06:40,547 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:06:40,547 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:06:40,547 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:40,746 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 17759 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 609478, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:40,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630472_1099518555102] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 5574 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 711280, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:06:40,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630473_1099518555103] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 2592 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692727, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:06:40,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 18704 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 514851, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:40,967 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:40,967 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:40,967 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:40,967 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:40,967 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:41,377 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630472_1099518555102] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 6082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853901, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:06:41,377 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 20488 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 558935, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:41,377 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630473_1099518555103] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 2989 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 858501, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:06:41,377 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:41,377 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:41,377 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:41,377 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:41,377 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:41,700 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#52 average throughput is 21.34 MB/sec, slept 6 time(s) and total slept time is 6845 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 13:06:41,898 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 23194 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 495354, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:41,898 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:41,898 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:41,898 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:41,898 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:41,898 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:41,956 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630473_1099518555103] hdfs.DFSClient: Slow ReadProcessor read fields took 253ms (threshold=20ms); ack: seqno: 3458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 252640296, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:06:41,956 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hdfs.DFSClient: Slow waitForAckedSeqno took 255ms (threshold=20ms) 2015-10-16 13:06:41,999 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1d384440601242c381fe7dbc44d3e1c0 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1d384440601242c381fe7dbc44d3e1c0 2015-10-16 13:06:42,012 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:06:42,016 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:06:42,024 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3939b430c93a4aaebfc46b09e2185db6, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3939b430c93a4aaebfc46b09e2185db6 2015-10-16 13:06:42,033 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5c0d7c395cea4f3fa017b0f71add0b9c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5c0d7c395cea4f3fa017b0f71add0b9c 2015-10-16 13:06:42,041 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/62ed602fa53b442b85fa521c619fff0b, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/62ed602fa53b442b85fa521c619fff0b 2015-10-16 13:06:42,041 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 1d384440601242c381fe7dbc44d3e1c0(size=212.5 M), total size for store is 16.0 G. This selection was in queue for 0sec, and took 10sec to execute. 2015-10-16 13:06:42,041 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=0, time=36903900851918131; duration=10sec 2015-10-16 13:06:42,041 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:2), split_queue=0, merge_queue=0 2015-10-16 13:06:42,041 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 4 compacting, 2 eligible, 10 blocking 2015-10-16 13:06:42,042 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:06:42,042 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:06:42,042 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:06:42,042 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 4 compacting, 2 eligible, 10 blocking 2015-10-16 13:06:42,043 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:06:42,043 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:06:42,043 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:06:42,380 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#51 average throughput is 17.86 MB/sec, slept 25 time(s) and total slept time is 16395 ms. 0 active compactions remaining, total limit is unlimited 2015-10-16 13:06:42,395 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/7b4a76b3e7f342de912e2ce0605e4ac6 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7b4a76b3e7f342de912e2ce0605e4ac6 2015-10-16 13:06:42,404 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:06:42,407 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:06:42,416 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7ab59b8a99514240b3cc1f59adfe9fed, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7ab59b8a99514240b3cc1f59adfe9fed 2015-10-16 13:06:42,424 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e98cdcf3ec40472e9cb6b35967503539, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e98cdcf3ec40472e9cb6b35967503539 2015-10-16 13:06:42,433 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/323aeea710f9476f835fee40c92e301d, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/323aeea710f9476f835fee40c92e301d 2015-10-16 13:06:42,441 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/192bffb1bf7b43d2ba75774f80e0c3d0, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/192bffb1bf7b43d2ba75774f80e0c3d0 2015-10-16 13:06:42,441 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 7b4a76b3e7f342de912e2ce0605e4ac6(size=425.3 M), total size for store is 16.0 G. This selection was in queue for 0sec, and took 23sec to execute. 2015-10-16 13:06:42,441 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=425.3 M, priority=3, time=36903888063167846; duration=23sec 2015-10-16 13:06:42,441 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:06:42,495 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 26312 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 493732, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:42,495 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:06:42,495 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:06:42,495 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:06:42,495 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:06:42,495 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:06:42,622 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=207025776 > lower bound=16777216 2015-10-16 13:06:42,622 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=197.41 MB 2015-10-16 13:06:42,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 27112 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 582464, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:43,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 30108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 426576, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:43,235 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:43,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630475_1099518555105] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 863 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1056824, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:43,235 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:43,235 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:43,799 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10847010, memsize=197.4 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/efc8e8d707fb486bb0906aee401e1583 2015-10-16 13:06:43,804 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/efc8e8d707fb486bb0906aee401e1583 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/efc8e8d707fb486bb0906aee401e1583 2015-10-16 13:06:43,811 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/efc8e8d707fb486bb0906aee401e1583, entries=718910, sequenceid=10847010, filesize=109.2 M 2015-10-16 13:06:43,813 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~197.45 MB/207039760, currentsize=33.56 MB/35192640 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1191ms, sequenceid=10847010, compaction requested=true 2015-10-16 13:06:43,813 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:06:43,813 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 0 compacting, 7 eligible, 10 blocking 2015-10-16 13:06:43,816 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 7 files of size 17275004236 starting at candidate #0 after considering 15 permutations with 5 in ratio 2015-10-16 13:06:43,816 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction (all files) 2015-10-16 13:06:43,816 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.RatioBasedCompactionPolicy: Selecting compaction from 7 store files, 0 compacting, 7 eligible, 10 blocking 2015-10-16 13:06:43,819 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 7 files of size 17275004236 starting at candidate #0 after considering 15 permutations with 5 in ratio 2015-10-16 13:06:43,819 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction (all files) 2015-10-16 13:06:43,819 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:06:43,819 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] regionserver.HStore: Starting compaction of 7 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=16.1 G 2015-10-16 13:06:43,820 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/40708f5c7dee4e848af4976f786425f6, keycount=57786180, bloomtype=ROW, size=8.6 G, encoding=NONE, seqNum=5778995, earliestPutTs=1445024396631 2015-10-16 13:06:43,820 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5c6694faa1b4472f963682d4511dbcf8, keycount=36772180, bloomtype=ROW, size=5.5 G, encoding=NONE, seqNum=9456437, earliestPutTs=1445025125930 2015-10-16 13:06:43,821 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/bd400d9a06d340e8a2bbd497821ef103, keycount=8521350, bloomtype=ROW, size=1.3 G, encoding=NONE, seqNum=10308621, earliestPutTs=1445025758875 2015-10-16 13:06:43,821 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7b4a76b3e7f342de912e2ce0605e4ac6, keycount=2799400, bloomtype=ROW, size=425.3 M, encoding=NONE, seqNum=10588586, earliestPutTs=1445025946189 2015-10-16 13:06:43,822 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1d384440601242c381fe7dbc44d3e1c0, keycount=1398810, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=10728480, earliestPutTs=1445025978354 2015-10-16 13:06:43,822 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3868aaf7bd7549fc9b596ac7877200c1, keycount=466290, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=10775113, earliestPutTs=1445025991052 2015-10-16 13:06:43,823 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/efc8e8d707fb486bb0906aee401e1583, keycount=718910, bloomtype=ROW, size=109.2 M, encoding=NONE, seqNum=10847010, earliestPutTs=1445025995419 2015-10-16 13:06:43,954 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 33764 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 585612, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:43,954 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:06:43,954 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:43,954 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:06:43,954 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:06:43,954 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:06:43,974 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:06:44,883 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 38939 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489263, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:44,883 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:06:44,883 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:06:44,883 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:06:44,883 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:06:44,883 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:06:44,885 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:06:44,923 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 13:06:44,966 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630471_1099518555101] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 39220 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 30388073, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:06:44,966 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:06:44,975 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025997216 with entries=78111, filesize=122.45 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026004886 2015-10-16 13:06:44,976 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025989807 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025989807 2015-10-16 13:06:44,978 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630477_1099518555107] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:45,081 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 97 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013291, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:45,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154799, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:45,296 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630477_1099518555107] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 1200 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 665874, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:45,344 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 134 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966955, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:45,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 178ms (threshold=20ms); ack: seqno: 146 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1114290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:45,572 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 158 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 831389, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:45,710 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 183 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913443, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:45,927 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630477_1099518555107] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 4780 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 543607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:45,927 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:45,927 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:45,927 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:45,927 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:45,927 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:46,104 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 385ms (threshold=20ms); ack: seqno: 195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1210837, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:46,323 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 212ms (threshold=20ms); ack: seqno: 207 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085939, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:46,425 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 219 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1191991, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:46,688 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 254ms (threshold=20ms); ack: seqno: 232 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070948, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:46,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 244 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1372836, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:46,844 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630477_1099518555107] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 10067 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 561124, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:46,844 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:46,844 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:46,844 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:46,844 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:46,844 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:47,020 DEBUG [PriorityRpcServer.handler=8,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:06:47,020 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134279536 > lower bound=16777216 2015-10-16 13:06:47,020 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:06:47,042 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630477_1099518555107] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 11125 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 618752, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:47,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 261ms (threshold=20ms); ack: seqno: 256 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1098717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:47,198 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011959, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:47,508 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 300ms (threshold=20ms); ack: seqno: 279 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1336364, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:47,574 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630477_1099518555107] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 13909 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 432919, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:47,574 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630478_1099518555108] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 738 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:06:47,574 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 305 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 705268, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:47,574 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:47,574 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:47,574 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:47,574 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:47,574 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:06:47,652 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 315 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:47,866 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10893641, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/e5489b12ea234e5eadcadf83c74a4126 2015-10-16 13:06:47,870 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/e5489b12ea234e5eadcadf83c74a4126 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e5489b12ea234e5eadcadf83c74a4126 2015-10-16 13:06:47,877 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e5489b12ea234e5eadcadf83c74a4126, entries=466270, sequenceid=10893641, filesize=70.8 M 2015-10-16 13:06:47,878 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134282000, currentsize=25.28 MB/26509840 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 858ms, sequenceid=10893641, compaction requested=false 2015-10-16 13:06:47,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 277ms (threshold=20ms); ack: seqno: 327 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983629, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:48,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 338 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022812, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:48,341 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 270ms (threshold=20ms); ack: seqno: 350 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1223317, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:48,379 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630477_1099518555107] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 18482 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 450959, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:48,379 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 356 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:48,379 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:48,379 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:48,379 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:48,379 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:48,379 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:48,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037227, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:48,478 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 367 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 831809, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:48,764 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 281ms (threshold=20ms); ack: seqno: 373 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1397213, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:48,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 384 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1425278, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:49,159 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 279ms (threshold=20ms); ack: seqno: 397 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1287499, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:49,220 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630477_1099518555107] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 22928 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 548265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:49,261 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 410 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1215926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:49,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630477_1099518555107] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 23800 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 478144, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:49,407 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:49,407 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:49,407 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:49,407 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:49,407 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:49,546 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 277ms (threshold=20ms); ack: seqno: 422 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1139369, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:49,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 434 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1082453, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:49,991 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 292ms (threshold=20ms); ack: seqno: 446 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1139654, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:50,085 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 459 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:50,357 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630477_1099518555107] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 29140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 670629, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:50,357 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:50,357 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:50,357 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:50,357 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:50,357 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:50,359 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 266ms (threshold=20ms); ack: seqno: 470 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174371, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:50,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 482 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1084647, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:50,705 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 250ms (threshold=20ms); ack: seqno: 494 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050867, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:50,844 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 500 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1281423, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:51,108 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 261ms (threshold=20ms); ack: seqno: 506 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1145507, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:51,232 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 519 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1168677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:51,298 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630477_1099518555107] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 34485 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 573245, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:51,299 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:51,299 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:51,299 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:06:51,299 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:06:51,299 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:06:51,299 DEBUG [PriorityRpcServer.handler=18,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:06:51,299 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134279392 > lower bound=16777216 2015-10-16 13:06:51,299 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.03 MB 2015-10-16 13:06:51,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 523 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1289105, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:51,322 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630477_1099518555107] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 34500 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 661793, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:51,380 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 531 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984454, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:51,461 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 67ms (threshold=20ms) 2015-10-16 13:06:51,461 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 67ms (threshold=20ms) 2015-10-16 13:06:51,461 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 67ms (threshold=20ms) 2015-10-16 13:06:51,461 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 67ms (threshold=20ms) 2015-10-16 13:06:51,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630477_1099518555107] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 34923 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:51,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 543 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946160, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:51,498 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 550 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845056, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:51,549 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 568 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 816115, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:51,584 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 579 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:51,797 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 178ms (threshold=20ms); ack: seqno: 603 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 925307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:51,874 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 795676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:51,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 627 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873504, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:52,058 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630479_1099518555109] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 830 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012444, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:52,058 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630477_1099518555107] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 37905 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 649177, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:06:52,058 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:52,058 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:52,058 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:52,058 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:52,058 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:52,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 639 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 856619, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:52,227 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:06:52,231 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 662 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813344, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:52,282 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10940279, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/f0fb0104076a4fc1992040bcdeaac88f 2015-10-16 13:06:52,287 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/f0fb0104076a4fc1992040bcdeaac88f as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f0fb0104076a4fc1992040bcdeaac88f 2015-10-16 13:06:52,289 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:52,293 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f0fb0104076a4fc1992040bcdeaac88f, entries=466340, sequenceid=10940279, filesize=70.8 M 2015-10-16 13:06:52,295 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134302880, currentsize=26.15 MB/27422320 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 995ms, sequenceid=10940279, compaction requested=false 2015-10-16 13:06:52,316 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026004886 with entries=78436, filesize=122.95 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026012227 2015-10-16 13:06:52,317 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025997216 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445025997216 2015-10-16 13:06:52,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630480_1099518555110] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1016858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:52,383 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 143ms (threshold=20ms); ack: seqno: 674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 852448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:52,545 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 698 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947935, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:52,755 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 202ms (threshold=20ms); ack: seqno: 711 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072914, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:52,894 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630480_1099518555110] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 3157 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 656230, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:52,894 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:06:52,894 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:06:52,894 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:06:52,894 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:06:52,894 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:06:52,895 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 735 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 897293, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:53,016 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 754 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 848960, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:53,138 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 765 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:53,172 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907879, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:53,290 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 782 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 900327, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:53,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 817 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932321, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:53,656 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 950 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943575, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:53,737 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630480_1099518555110] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 7809 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 561114, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:53,737 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:06:53,737 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:06:53,738 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:06:53,738 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:06:53,738 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:06:53,757 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 963 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1091028, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:53,965 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 1106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 892378, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:54,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 1117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 826071, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:54,199 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 1138 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1079363, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:54,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630480_1099518555110] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 9985 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 757499, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:54,316 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 1149 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 831108, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:54,529 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 1292 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1055505, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:54,597 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630480_1099518555110] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 11995 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 394984, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:54,598 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:54,598 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:54,598 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:54,598 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:54,598 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:54,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 1306 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:54,752 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 1318 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 871713, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:54,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 1342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:54,982 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 1353 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974083, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:55,095 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 1366 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 899858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:55,209 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 1377 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853815, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:55,325 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 1388 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838895, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:55,441 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 1400 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885468, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:55,486 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:55,486 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:55,487 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630480_1099518555110] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 17144 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 710515, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:55,487 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 1412 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 951740, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:55,599 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 1417 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 810040, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:55,732 DEBUG [PriorityRpcServer.handler=14,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:06:55,732 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134288864 > lower bound=16777216 2015-10-16 13:06:55,732 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 13:06:55,757 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630480_1099518555110] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 18627 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 570155, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:56,029 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 422ms (threshold=20ms); ack: seqno: 1429 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188775, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:56,189 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 152ms (threshold=20ms); ack: seqno: 1441 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1031351, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:56,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630480_1099518555110] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 21277 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 463127, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:56,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630481_1099518555111] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 649 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 720467, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:06:56,244 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:56,244 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:56,244 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:56,244 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:56,245 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:06:56,406 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 208ms (threshold=20ms); ack: seqno: 1454 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1067026, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:56,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 1466 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1033198, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:56,582 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=10986913, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/29e5276f5857406b8e1d6c6001533aab 2015-10-16 13:06:56,587 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/29e5276f5857406b8e1d6c6001533aab as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/29e5276f5857406b8e1d6c6001533aab 2015-10-16 13:06:56,593 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/29e5276f5857406b8e1d6c6001533aab, entries=466300, sequenceid=10986913, filesize=70.8 M 2015-10-16 13:06:56,595 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134289600, currentsize=25.75 MB/27005280 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 863ms, sequenceid=10986913, compaction requested=true 2015-10-16 13:06:56,595 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:06:56,595 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 7 compacting, 3 eligible, 10 blocking 2015-10-16 13:06:56,596 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 222842483 starting at candidate #0 after considering 1 permutations with 1 in ratio 2015-10-16 13:06:56,596 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:06:56,596 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:06:56,596 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=212.5 M 2015-10-16 13:06:56,596 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e5489b12ea234e5eadcadf83c74a4126, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=10893641 2015-10-16 13:06:56,597 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f0fb0104076a4fc1992040bcdeaac88f, keycount=466340, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=10940279 2015-10-16 13:06:56,597 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/29e5276f5857406b8e1d6c6001533aab, keycount=466300, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=10986913 2015-10-16 13:06:56,611 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:06:56,663 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 1577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 976193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:56,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630482_1099518555112] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 854262, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:56,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630480_1099518555110] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 24048 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 458276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:56,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 1677 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 824540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:56,781 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:56,781 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:56,781 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:56,781 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:56,781 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:56,955 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 164ms (threshold=20ms); ack: seqno: 1687 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4404659, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:57,227 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 268ms (threshold=20ms); ack: seqno: 1700 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1117645, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:57,307 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630480_1099518555110] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 26786 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 558385, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:57,307 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630482_1099518555112] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 763237, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:57,307 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:57,307 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:57,307 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:57,307 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:57,307 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:57,334 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 1711 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019560, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:57,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 266ms (threshold=20ms); ack: seqno: 1723 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1249914, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:57,723 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630480_1099518555110] hdfs.DFSClient: Slow ReadProcessor read fields took 217ms (threshold=20ms); ack: seqno: 27833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 761162, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:57,724 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 217ms (threshold=20ms) 2015-10-16 13:06:57,724 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 217ms (threshold=20ms) 2015-10-16 13:06:57,724 INFO [sync.1] wal.FSHLog: Slow sync cost: 217 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:57,724 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 217ms (threshold=20ms) 2015-10-16 13:06:57,724 INFO [sync.2] wal.FSHLog: Slow sync cost: 217 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:57,724 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 217ms (threshold=20ms) 2015-10-16 13:06:57,724 INFO [sync.4] wal.FSHLog: Slow sync cost: 217 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:57,724 INFO [sync.0] wal.FSHLog: Slow sync cost: 217 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:57,724 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 217ms (threshold=20ms) 2015-10-16 13:06:57,724 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 1724 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1206760, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:57,724 INFO [sync.3] wal.FSHLog: Slow sync cost: 217 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:57,919 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630482_1099518555112] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 1681 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:57,919 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630480_1099518555110] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 28753 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 466379, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:57,919 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:57,919 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:57,919 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:57,919 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:57,919 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:06:58,020 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 290ms (threshold=20ms); ack: seqno: 1747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1280283, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:58,155 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 1760 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102647, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:58,211 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630482_1099518555112] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 2082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 21546124, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:06:58,445 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 282ms (threshold=20ms); ack: seqno: 1771 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1173775, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:58,527 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630480_1099518555110] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 32153 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 381326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:58,527 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:58,527 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:58,527 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:58,530 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:58,530 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:06:58,547 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 1782 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1201044, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:58,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 310ms (threshold=20ms); ack: seqno: 1794 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050112, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:58,978 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 1805 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1131595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:59,287 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 301ms (threshold=20ms); ack: seqno: 1817 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1180844, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:59,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630480_1099518555110] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 37588 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 447592, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:06:59,443 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:59,443 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:59,443 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:59,443 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:06:59,443 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:06:59,448 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 153ms (threshold=20ms); ack: seqno: 1829 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1130290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:59,647 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:06:59,698 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 243ms (threshold=20ms); ack: seqno: 1841 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:06:59,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 153ms (threshold=20ms); ack: seqno: 1852 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1244237, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:07:00,925 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 1043ms (threshold=20ms); ack: seqno: 1865 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 781464620, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:07:01,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 171ms (threshold=20ms); ack: seqno: 1920 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1170147, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:07:01,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 1926 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1139866, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:07:01,429 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 262ms (threshold=20ms); ack: seqno: 1933 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1633110, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:07:01,487 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 1940 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1131467, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:07:01,598 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 1951 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926785, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:07:01,722 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630476_1099518555106] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 2082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 952276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:07:01,722 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630480_1099518555110] hdfs.DFSClient: Slow ReadProcessor read fields took 1727ms (threshold=20ms); ack: seqno: 41035 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 47751891, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:01,932 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 1938ms (threshold=20ms) 2015-10-16 13:07:01,932 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 1939ms (threshold=20ms) 2015-10-16 13:07:01,932 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 2275ms (threshold=20ms) 2015-10-16 13:07:01,932 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 1938ms (threshold=20ms) 2015-10-16 13:07:01,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630483_1099518555113] hdfs.DFSClient: Slow ReadProcessor read fields took 153ms (threshold=20ms); ack: seqno: 2194 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3669398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:01,933 INFO [sync.3] wal.FSHLog: Slow sync cost: 1938 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:01,932 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 1938ms (threshold=20ms) 2015-10-16 13:07:01,932 INFO [sync.0] wal.FSHLog: Slow sync cost: 1938 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:01,932 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 1938ms (threshold=20ms) 2015-10-16 13:07:01,933 INFO [sync.2] wal.FSHLog: Slow sync cost: 1938 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:01,933 INFO [sync.4] wal.FSHLog: Slow sync cost: 1938 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:01,940 INFO [sync.1] wal.FSHLog: Slow sync cost: 1945 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:01,948 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Slow sync cost: 2291 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:01,957 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026012227 with entries=81755, filesize=128.16 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026019647 2015-10-16 13:07:01,959 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026004886 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026004886 2015-10-16 13:07:01,960 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630484_1099518555114] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1056174, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:02,038 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630484_1099518555114] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 203 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 592795, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:02,038 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630483_1099518555113] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 2301 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 799998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:02,038 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:07:02,038 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:02,038 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:07:02,038 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:07:02,038 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:07:02,234 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630484_1099518555114] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 591 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 636309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:02,259 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 2176 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1399682, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:02,302 DEBUG [PriorityRpcServer.handler=7,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:07:02,302 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134274832 > lower bound=16777216 2015-10-16 13:07:02,302 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:07:02,327 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630484_1099518555114] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 968 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 492083, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:02,575 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630487_1099518555117] hdfs.DFSClient: Slow ReadProcessor read fields took 217ms (threshold=20ms); ack: seqno: 18 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 202224188, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:02,575 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630483_1099518555113] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 3066 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 716944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:02,575 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630484_1099518555114] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 2161 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 492825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:02,575 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:02,575 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:02,575 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:02,575 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:02,575 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:02,578 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 310ms (threshold=20ms); ack: seqno: 2189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1433350, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:02,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 2200 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013096, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:02,743 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 13:07:02,743 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 13:07:02,743 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 13:07:02,841 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#54 average throughput is 33.19 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 13:07:02,853 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/7e0d37c25a5247a6845e5cd97223d021 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7e0d37c25a5247a6845e5cd97223d021 2015-10-16 13:07:02,861 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:07:02,865 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:07:02,873 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e5489b12ea234e5eadcadf83c74a4126, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/e5489b12ea234e5eadcadf83c74a4126 2015-10-16 13:07:02,881 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f0fb0104076a4fc1992040bcdeaac88f, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/f0fb0104076a4fc1992040bcdeaac88f 2015-10-16 13:07:02,890 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/29e5276f5857406b8e1d6c6001533aab, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/29e5276f5857406b8e1d6c6001533aab 2015-10-16 13:07:02,890 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 7e0d37c25a5247a6845e5cd97223d021(size=212.5 M), total size for store is 16.3 G. This selection was in queue for 0sec, and took 6sec to execute. 2015-10-16 13:07:02,890 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=212.5 M, priority=0, time=36903925450357864; duration=6sec 2015-10-16 13:07:02,890 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:07:02,947 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 279ms (threshold=20ms); ack: seqno: 2214 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1137865, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:03,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630487_1099518555117] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 740 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865033, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:03,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630484_1099518555114] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 4354 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 604939, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:03,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 2285 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836921, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:03,030 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 13:07:03,030 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 13:07:03,030 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 13:07:03,030 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 13:07:03,030 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 13:07:03,118 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 2297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1074799, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:03,423 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 297ms (threshold=20ms); ack: seqno: 2309 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1425945, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:03,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 2321 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1473743, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:03,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630487_1099518555117] hdfs.DFSClient: Slow ReadProcessor read fields took 460ms (threshold=20ms); ack: seqno: 1031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1297915, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:03,740 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11033544, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/98bbaac1d8eb497ba7fe299dd060b121 2015-10-16 13:07:03,744 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/98bbaac1d8eb497ba7fe299dd060b121 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/98bbaac1d8eb497ba7fe299dd060b121 2015-10-16 13:07:03,751 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/98bbaac1d8eb497ba7fe299dd060b121, entries=466270, sequenceid=11033544, filesize=70.8 M 2015-10-16 13:07:03,752 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134282480, currentsize=42.68 MB/44756000 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1450ms, sequenceid=11033544, compaction requested=false 2015-10-16 13:07:03,832 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630484_1099518555114] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 9013 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 312555, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:03,832 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:07:03,832 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:07:03,832 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:07:03,832 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:07:03,832 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:07:03,834 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 294ms (threshold=20ms); ack: seqno: 2332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1396668, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:03,926 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 2345 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1227414, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:04,050 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 2358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3610637, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:04,338 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 273ms (threshold=20ms); ack: seqno: 2370 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1420103, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:04,452 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 2382 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1288431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:04,719 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630484_1099518555114] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 14309 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 337637, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:04,719 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:04,719 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:04,719 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:04,719 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:04,720 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:04,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 274ms (threshold=20ms); ack: seqno: 2394 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1369477, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:04,867 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 2406 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1766096, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:05,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 278ms (threshold=20ms); ack: seqno: 2419 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1574166, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:05,378 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 220ms (threshold=20ms); ack: seqno: 2430 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1521203, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:05,561 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 176ms (threshold=20ms); ack: seqno: 2441 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1310069, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:05,640 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630484_1099518555114] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 19660 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 444791, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:05,640 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:05,640 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:05,669 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 2453 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:05,984 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 307ms (threshold=20ms); ack: seqno: 2466 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1248251, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:06,170 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 182ms (threshold=20ms); ack: seqno: 2474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1344991, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:06,370 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 194ms (threshold=20ms); ack: seqno: 2483 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1483977, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:06,386 DEBUG [PriorityRpcServer.handler=16,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:07:06,386 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134273744 > lower bound=16777216 2015-10-16 13:07:06,386 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:07:06,415 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630484_1099518555114] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 24377 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 529757, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:06,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630484_1099518555114] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 24834 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 624716, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:06,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 2496 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7325496, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:06,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630488_1099518555118] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 99 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 771308, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:07:06,530 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:07:06,530 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:07:06,530 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:07:06,530 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:07:06,530 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:07:06,773 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 230ms (threshold=20ms); ack: seqno: 2508 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1219803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:06,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 2525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5744496, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:06,985 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 2533 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1080164, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:07,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 2548 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1138725, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:07,179 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630484_1099518555114] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 28163 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 483189, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:07,179 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630488_1099518555118] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 1119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1577845, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:07:07,179 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:07,179 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:07,180 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:07,180 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:07,180 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:07:07,181 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 2566 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1475983, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:07,206 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11080175, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/0efa6666039a4b69890d685b45aced19 2015-10-16 13:07:07,212 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/0efa6666039a4b69890d685b45aced19 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0efa6666039a4b69890d685b45aced19 2015-10-16 13:07:07,217 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0efa6666039a4b69890d685b45aced19, entries=466260, sequenceid=11080175, filesize=70.8 M 2015-10-16 13:07:07,218 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134277360, currentsize=22.04 MB/23111440 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 832ms, sequenceid=11080175, compaction requested=true 2015-10-16 13:07:07,218 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:07:07,218 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 7 compacting, 3 eligible, 10 blocking 2015-10-16 13:07:07,220 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 13:07:07,220 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:07:07,220 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:07:07,288 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 2586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1056273, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:07,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 2619 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:07,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 226ms (threshold=20ms); ack: seqno: 2621 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3934777, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:07,758 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 2634 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1185923, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:07,776 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630484_1099518555114] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 30865 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836828, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:07,830 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 2646 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1004519, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:08,052 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 2781 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1065297, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:08,113 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630484_1099518555114] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 32687 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 434745, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:08,113 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:08,113 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:08,113 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:08,113 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:08,113 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:08,322 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 262ms (threshold=20ms); ack: seqno: 2792 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1078155, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:08,434 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 2804 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:08,523 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 2834 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012710, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:08,662 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 2874 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5909169, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:08,760 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 2886 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:08,984 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630484_1099518555114] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 37829 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 511935, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:08,984 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:08,984 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:08,984 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:08,984 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:08,984 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:09,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 281ms (threshold=20ms); ack: seqno: 2898 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1388659, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:09,168 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:07:09,212 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:09,223 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026019647 with entries=78149, filesize=122.50 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026029168 2015-10-16 13:07:09,224 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026012227 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026012227 2015-10-16 13:07:09,416 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 360ms (threshold=20ms); ack: seqno: 2910 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1191255, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:09,522 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 2922 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1312190, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:09,813 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 284ms (threshold=20ms); ack: seqno: 2935 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1172446, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:09,929 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630489_1099518555119] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 3967 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 449640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:09,929 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:09,929 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:09,929 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:09,929 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:09,929 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:09,930 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 2946 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1229758, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:10,201 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 262ms (threshold=20ms); ack: seqno: 2958 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1121178, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:10,326 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 2969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022094, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:10,698 DEBUG [PriorityRpcServer.handler=6,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:07:10,698 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134284976 > lower bound=16777216 2015-10-16 13:07:10,698 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 13:07:10,721 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 387ms (threshold=20ms); ack: seqno: 2982 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1342545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:10,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630489_1099518555119] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 8565 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 614769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:10,772 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 2994 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965332, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:10,863 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630489_1099518555119] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 9023 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 456424, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:10,863 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 54ms (threshold=20ms) 2015-10-16 13:07:10,863 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 53ms (threshold=20ms) 2015-10-16 13:07:10,863 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630490_1099518555120] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 95 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 884002, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:07:10,863 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 53ms (threshold=20ms) 2015-10-16 13:07:10,863 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 53ms (threshold=20ms) 2015-10-16 13:07:10,918 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 3012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 923577, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:11,219 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 291ms (threshold=20ms); ack: seqno: 3023 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914486, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:11,329 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 3035 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1118719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:11,519 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:11,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630490_1099518555120] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 1028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 823911, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:07:11,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630489_1099518555119] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 12643 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 460771, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:11,519 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:11,519 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:11,519 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:11,519 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:11,606 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11126809, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/199c3eaa20624bc08168f49159dca781 2015-10-16 13:07:11,610 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/199c3eaa20624bc08168f49159dca781 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/199c3eaa20624bc08168f49159dca781 2015-10-16 13:07:11,617 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/199c3eaa20624bc08168f49159dca781, entries=466300, sequenceid=11126809, filesize=70.8 M 2015-10-16 13:07:11,618 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134290320, currentsize=25.23 MB/26455360 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 920ms, sequenceid=11126809, compaction requested=true 2015-10-16 13:07:11,618 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:07:11,618 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 7 compacting, 4 eligible, 10 blocking 2015-10-16 13:07:11,620 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 445674075 starting at candidate #0 after considering 3 permutations with 2 in ratio 2015-10-16 13:07:11,620 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:07:11,620 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:07:11,620 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=425.0 M 2015-10-16 13:07:11,620 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7e0d37c25a5247a6845e5cd97223d021, keycount=1398910, bloomtype=ROW, size=212.5 M, encoding=NONE, seqNum=10986913 2015-10-16 13:07:11,620 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/98bbaac1d8eb497ba7fe299dd060b121, keycount=466270, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=11033544 2015-10-16 13:07:11,620 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0efa6666039a4b69890d685b45aced19, keycount=466260, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=11080175 2015-10-16 13:07:11,620 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/199c3eaa20624bc08168f49159dca781, keycount=466300, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=11126809 2015-10-16 13:07:11,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 285ms (threshold=20ms); ack: seqno: 3049 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:11,635 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:07:11,683 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 3054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1133975, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:11,984 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 294ms (threshold=20ms); ack: seqno: 3065 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1115981, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:12,134 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630489_1099518555119] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 15432 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 513290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:12,136 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 85ms (threshold=20ms) 2015-10-16 13:07:12,136 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 85ms (threshold=20ms) 2015-10-16 13:07:12,136 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 86ms (threshold=20ms) 2015-10-16 13:07:12,136 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 86ms (threshold=20ms) 2015-10-16 13:07:12,136 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 86ms (threshold=20ms) 2015-10-16 13:07:12,136 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630491_1099518555121] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 560 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072995, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:12,137 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 148ms (threshold=20ms); ack: seqno: 3073 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1464027, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:12,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630491_1099518555121] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 1238 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 688469, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:12,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630489_1099518555119] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 18172 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 473769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:12,659 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:12,659 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:12,659 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:12,660 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:12,660 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:12,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 543ms (threshold=20ms); ack: seqno: 3083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062371, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:12,946 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 249ms (threshold=20ms); ack: seqno: 3097 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1249811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:13,221 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630491_1099518555121] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 2001 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 614584, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:13,223 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630489_1099518555119] hdfs.DFSClient: Slow ReadProcessor read fields took 154ms (threshold=20ms); ack: seqno: 20508 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898849, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:13,223 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 270ms (threshold=20ms); ack: seqno: 3108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1177214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:13,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 227ms (threshold=20ms); ack: seqno: 3120 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1313863, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:13,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 328ms (threshold=20ms); ack: seqno: 3133 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127163, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:13,839 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 3158 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 770069, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:13,883 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630492_1099518555122] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 2419 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 931102, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:13,883 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630489_1099518555119] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 24104 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 436256, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:13,883 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:07:13,883 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:07:13,883 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:07:13,883 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:07:13,884 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:07:13,974 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 3170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1232631, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:14,143 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 3211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1045252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:14,376 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630492_1099518555122] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 3080 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 724298, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:14,376 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630489_1099518555119] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 26689 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1324538, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:14,376 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:07:14,376 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:07:14,376 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:07:14,377 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:07:14,377 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:14,378 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 232ms (threshold=20ms); ack: seqno: 3215 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020138, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:14,578 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 195ms (threshold=20ms); ack: seqno: 3223 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1341769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:14,650 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 3235 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1095294, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:14,693 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 3246 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 878580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:14,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 3259 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:14,869 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630489_1099518555119] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 29291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 490702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:14,869 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630492_1099518555122] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 3747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 738380, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:14,869 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:07:14,869 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:07:14,869 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:07:14,869 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:07:14,870 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 3271 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917288, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:14,980 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 3282 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950219, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:15,008 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 3289 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887709, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:15,043 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 3301 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:15,196 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 145ms (threshold=20ms); ack: seqno: 3313 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:15,317 DEBUG [PriorityRpcServer.handler=6,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:07:15,317 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 13:07:15,354 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630493_1099518555123] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 4413 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 644745, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:07:15,354 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630489_1099518555119] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 31862 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 494960, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:15,354 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:07:15,354 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:15,354 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:15,354 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:15,354 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:15,354 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:15,366 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 140ms (threshold=20ms); ack: seqno: 3331 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1138345, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:15,538 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 160ms (threshold=20ms); ack: seqno: 3343 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5126314, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:15,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 173ms (threshold=20ms); ack: seqno: 3361 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908384, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:15,762 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 3372 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 876945, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:15,837 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630493_1099518555123] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 5066 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 709615, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:07:15,837 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630489_1099518555119] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 34476 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 518577, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:15,837 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:15,837 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:15,837 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:15,837 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:15,898 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 3386 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896082, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:16,059 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 151ms (threshold=20ms); ack: seqno: 3398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1118353, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:16,219 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 151ms (threshold=20ms); ack: seqno: 3410 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070200, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:16,322 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630489_1099518555119] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 37141 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 484621, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:16,322 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630493_1099518555123] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 5729 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 700620, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:07:16,322 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:07:16,322 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:16,322 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:16,322 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:16,322 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:16,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 227ms (threshold=20ms); ack: seqno: 3421 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1205559, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:16,621 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:07:16,622 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 3440 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:16,819 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630489_1099518555119] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 39747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 526894, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:16,819 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630495_1099518555125] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 6411 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 735143, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:16,819 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:07:16,819 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:16,819 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:16,819 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:16,819 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:16,835 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 203ms (threshold=20ms) 2015-10-16 13:07:16,839 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Slow sync cost: 207 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:16,841 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 211ms (threshold=20ms); ack: seqno: 3451 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1156959, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:16,848 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026029168 with entries=79608, filesize=124.80 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026036621 2015-10-16 13:07:16,851 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026019647 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026019647 2015-10-16 13:07:16,985 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 3463 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:17,102 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 3470 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873884, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:17,175 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#55 average throughput is 74.65 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 13:07:17,194 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/4d6d4c12178e4c909876d90556b9c9c1 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4d6d4c12178e4c909876d90556b9c9c1 2015-10-16 13:07:17,201 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:07:17,205 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:07:17,214 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7e0d37c25a5247a6845e5cd97223d021, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7e0d37c25a5247a6845e5cd97223d021 2015-10-16 13:07:17,222 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/98bbaac1d8eb497ba7fe299dd060b121, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/98bbaac1d8eb497ba7fe299dd060b121 2015-10-16 13:07:17,230 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0efa6666039a4b69890d685b45aced19, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/0efa6666039a4b69890d685b45aced19 2015-10-16 13:07:17,239 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/199c3eaa20624bc08168f49159dca781, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/199c3eaa20624bc08168f49159dca781 2015-10-16 13:07:17,239 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 4d6d4c12178e4c909876d90556b9c9c1(size=425.0 M), total size for store is 16.5 G. This selection was in queue for 0sec, and took 5sec to execute. 2015-10-16 13:07:17,239 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=425.0 M, priority=-1, time=36903940473882452; duration=5sec 2015-10-16 13:07:17,239 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:07:17,239 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 7 compacting, 1 eligible, 10 blocking 2015-10-16 13:07:17,240 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:07:17,240 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:07:17,240 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:07:17,247 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 3481 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835070, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:17,373 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 3486 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942116, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:17,439 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630494_1099518555124] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 3181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 487246, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:17,440 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:07:17,440 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:07:17,440 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:07:17,440 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:07:17,440 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:07:17,482 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 3493 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 837582, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:17,539 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 3499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:17,591 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 3509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864931, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:17,642 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 3515 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 903980, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:17,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 3521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:17,882 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 3552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1190278, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:17,918 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630494_1099518555124] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 5322 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 766861, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:18,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 3564 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 903708, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:18,054 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=214783792 > lower bound=16777216 2015-10-16 13:07:18,055 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=204.78 MB 2015-10-16 13:07:18,096 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630494_1099518555124] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 6096 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842196, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:18,302 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 275ms (threshold=20ms); ack: seqno: 3576 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919219, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:18,427 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630494_1099518555124] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 7622 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 475957, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:18,427 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:18,428 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:18,428 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:18,429 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 3589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047632, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:18,680 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630494_1099518555124] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 8346 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1086724, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:18,697 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 262ms (threshold=20ms); ack: seqno: 3600 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010269, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:18,737 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630496_1099518555126] hdfs.DFSClient: Slow ReadProcessor read fields took 438ms (threshold=20ms); ack: seqno: 301 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1084984, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:07:18,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 3613 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1251271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:19,101 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 292ms (threshold=20ms); ack: seqno: 3624 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946381, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:19,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 3636 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 980370, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:19,309 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630496_1099518555126] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 1188 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736388, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:07:19,309 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630494_1099518555124] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 11431 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 496398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:19,309 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:19,309 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:19,309 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:19,309 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:19,497 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 257ms (threshold=20ms); ack: seqno: 3648 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044782, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:19,732 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 224ms (threshold=20ms); ack: seqno: 3659 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1134552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:19,896 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 157ms (threshold=20ms); ack: seqno: 3672 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174961, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:19,997 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 3684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054009, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:20,065 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630494_1099518555124] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 15527 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 451764, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:20,065 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:07:20,065 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:20,066 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:20,065 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:07:20,066 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:20,263 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 258ms (threshold=20ms); ack: seqno: 3696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:20,414 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 3707 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006951, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:20,516 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 3721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787947, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:20,697 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630496_1099518555126] hdfs.DFSClient: Slow ReadProcessor read fields took 967ms (threshold=20ms); ack: seqno: 1843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1071772, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:07:20,697 WARN [MemStoreFlusher.0] hdfs.DFSClient: Slow waitForAckedSeqno took 968ms (threshold=20ms) 2015-10-16 13:07:20,705 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11201396, memsize=204.8 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5262b65b84ab44e980f2c1f9b79d0074 2015-10-16 13:07:20,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5262b65b84ab44e980f2c1f9b79d0074 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5262b65b84ab44e980f2c1f9b79d0074 2015-10-16 13:07:20,716 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5262b65b84ab44e980f2c1f9b79d0074, entries=745820, sequenceid=11201396, filesize=113.3 M 2015-10-16 13:07:20,722 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~204.84 MB/214790000, currentsize=73.10 MB/76648800 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 2668ms, sequenceid=11201396, compaction requested=false 2015-10-16 13:07:20,828 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 304ms (threshold=20ms); ack: seqno: 3732 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2622381, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:20,870 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 3744 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841741, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:20,985 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630494_1099518555124] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 20866 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 495237, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:20,985 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:20,985 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:20,985 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:20,986 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:20,985 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:21,214 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 337ms (threshold=20ms); ack: seqno: 3755 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1197570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:21,298 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 3766 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989199, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:21,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 294ms (threshold=20ms); ack: seqno: 3778 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956049, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:21,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 3790 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1049379, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:21,870 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630494_1099518555124] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 26224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 400385, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:21,870 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:21,870 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:21,870 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:21,870 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:21,870 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:22,062 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 304ms (threshold=20ms); ack: seqno: 3802 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1225089, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:22,554 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630494_1099518555124] hdfs.DFSClient: Slow ReadProcessor read fields took 571ms (threshold=20ms); ack: seqno: 26952 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:22,554 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 571ms (threshold=20ms) 2015-10-16 13:07:22,554 INFO [sync.0] wal.FSHLog: Slow sync cost: 570 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:22,554 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 572ms (threshold=20ms) 2015-10-16 13:07:22,554 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 571ms (threshold=20ms) 2015-10-16 13:07:22,554 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 572ms (threshold=20ms) 2015-10-16 13:07:22,554 INFO [sync.1] wal.FSHLog: Slow sync cost: 571 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:22,554 INFO [sync.3] wal.FSHLog: Slow sync cost: 571 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:22,554 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 571ms (threshold=20ms) 2015-10-16 13:07:22,554 INFO [sync.4] wal.FSHLog: Slow sync cost: 571 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:22,554 INFO [sync.2] wal.FSHLog: Slow sync cost: 571 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:22,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 492ms (threshold=20ms); ack: seqno: 3803 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2048776, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:22,700 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 3895 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921433, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:22,785 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 3907 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 826972, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:22,812 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 3908 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 747929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:22,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 3918 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 826776, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:22,993 DEBUG [PriorityRpcServer.handler=1,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:07:22,993 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134259584 > lower bound=16777216 2015-10-16 13:07:22,993 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:07:23,094 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 222ms (threshold=20ms); ack: seqno: 3929 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 929725, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:23,176 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 3940 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 886072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:23,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630497_1099518555127] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 274 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 665146, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:23,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630494_1099518555124] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 30594 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 438227, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:23,245 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:23,245 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:23,245 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:23,245 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:23,245 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:23,251 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 3952 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 968810, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:23,387 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 4060 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 849568, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:23,544 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630486_1099518555116] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 4161 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907228, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:07:23,633 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 4177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1382552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:23,814 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630497_1099518555127] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 1069 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827511, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:23,814 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630494_1099518555124] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 33636 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 588434, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:23,814 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:23,814 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:23,814 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:23,814 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:23,815 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:07:23,845 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 203ms (threshold=20ms); ack: seqno: 4189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1298176, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:23,872 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11248024, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/294b1f8242bf4f74bdda9690f28aef4a 2015-10-16 13:07:23,876 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/294b1f8242bf4f74bdda9690f28aef4a as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/294b1f8242bf4f74bdda9690f28aef4a 2015-10-16 13:07:23,883 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/294b1f8242bf4f74bdda9690f28aef4a, entries=466240, sequenceid=11248024, filesize=70.8 M 2015-10-16 13:07:23,883 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134273280, currentsize=24.62 MB/25812800 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 890ms, sequenceid=11248024, compaction requested=true 2015-10-16 13:07:23,884 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:07:23,884 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 7 compacting, 3 eligible, 10 blocking 2015-10-16 13:07:23,885 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 13:07:23,885 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:07:23,885 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:07:23,928 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 4200 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1518391, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:24,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 4212 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1135565, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:24,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630494_1099518555124] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 34809 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779892, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:24,231 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 216ms (threshold=20ms); ack: seqno: 4224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1165590, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:24,325 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 4236 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1009142, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:24,418 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 4248 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1029376, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:24,651 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 226ms (threshold=20ms); ack: seqno: 4260 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1121306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:24,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 4270 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1077448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:24,801 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630494_1099518555124] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 38640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 571274, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:24,801 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:24,801 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:24,801 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:24,801 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:24,801 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:24,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 4281 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1080656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:24,870 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:07:25,069 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 225ms (threshold=20ms); ack: seqno: 4293 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020745, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:25,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 4298 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 951867, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:25,192 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 4305 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:25,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 4318 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956167, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:25,377 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 4331 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043462, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:25,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 221ms (threshold=20ms); ack: seqno: 4342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1478070, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:25,764 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 151ms (threshold=20ms); ack: seqno: 4355 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1235127, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:25,805 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 4361 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:25,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 4366 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964373, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:26,052 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 140ms (threshold=20ms); ack: seqno: 4493 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022303, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:26,143 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630494_1099518555124] hdfs.DFSClient: Slow ReadProcessor read fields took 931ms (threshold=20ms); ack: seqno: 41080 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2583156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:07:26,156 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 1276ms (threshold=20ms) 2015-10-16 13:07:26,178 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 967ms (threshold=20ms) 2015-10-16 13:07:26,179 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 967ms (threshold=20ms) 2015-10-16 13:07:26,179 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 968ms (threshold=20ms) 2015-10-16 13:07:26,179 INFO [sync.1] wal.FSHLog: Slow sync cost: 967 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:07:26,179 INFO [sync.3] wal.FSHLog: Slow sync cost: 967 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:07:26,179 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 967ms (threshold=20ms) 2015-10-16 13:07:26,179 INFO [sync.4] wal.FSHLog: Slow sync cost: 967 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:07:26,179 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 967ms (threshold=20ms) 2015-10-16 13:07:26,179 INFO [sync.0] wal.FSHLog: Slow sync cost: 967 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:07:26,205 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Slow sync cost: 1324 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:07:26,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630500_1099518555130] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 41090 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:07:26,224 INFO [sync.2] wal.FSHLog: Slow sync cost: 1010 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:26,224 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026036621 with entries=81676, filesize=128.03 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026044870 2015-10-16 13:07:26,227 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026029168 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026029168 2015-10-16 13:07:26,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1169142, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:26,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 185ms (threshold=20ms); ack: seqno: 4504 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943352, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:26,471 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 220ms (threshold=20ms); ack: seqno: 4516 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1123794, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:26,578 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 2079 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 490342, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:26,578 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:26,578 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:26,578 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:26,578 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:26,578 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:26,580 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 4528 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1171444, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:26,790 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 203ms (threshold=20ms); ack: seqno: 4540 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062118, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:26,877 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 4552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 958481, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:26,962 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 4564 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 903941, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:27,199 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 230ms (threshold=20ms); ack: seqno: 4576 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1075454, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:27,281 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 4588 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 933601, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:27,356 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 4602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792934, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:27,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 7382 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 547519, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:27,431 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:27,431 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:27,431 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:27,431 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:27,573 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 209ms (threshold=20ms); ack: seqno: 4615 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1040244, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:27,651 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 4626 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:27,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 4637 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:27,953 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 206ms (threshold=20ms); ack: seqno: 4649 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 949633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:28,032 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 4662 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 882811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:28,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 4673 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797578, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:28,211 DEBUG [PriorityRpcServer.handler=0,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:07:28,212 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134301376 > lower bound=16777216 2015-10-16 13:07:28,212 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.03 MB 2015-10-16 13:07:28,242 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 11981 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 653893, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:28,365 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 12422 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 465666, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:28,365 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630501_1099518555131] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 61 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2491615, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:28,365 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:07:28,365 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:07:28,365 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:07:28,365 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:07:28,365 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:07:28,367 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 240ms (threshold=20ms); ack: seqno: 4685 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1299976, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:28,448 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 4697 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905181, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:28,525 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 4708 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:28,760 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 227ms (threshold=20ms); ack: seqno: 4721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:28,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 4734 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:28,956 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 4746 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874818, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:29,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 16091 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 419731, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:29,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630501_1099518555131] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 897 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 685847, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:29,030 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:29,030 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:29,030 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:29,030 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:29,030 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:29,174 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 209ms (threshold=20ms); ack: seqno: 4757 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1045438, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:29,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11294664, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/94288a9a9ed943649614a169205dae35 2015-10-16 13:07:29,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/94288a9a9ed943649614a169205dae35 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/94288a9a9ed943649614a169205dae35 2015-10-16 13:07:29,224 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/94288a9a9ed943649614a169205dae35, entries=466360, sequenceid=11294664, filesize=70.8 M 2015-10-16 13:07:29,226 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.08 MB/134306720, currentsize=29.20 MB/30622400 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1014ms, sequenceid=11294664, compaction requested=true 2015-10-16 13:07:29,226 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:07:29,226 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 7 compacting, 4 eligible, 10 blocking 2015-10-16 13:07:29,228 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 3 files of size 267391832 starting at candidate #1 after considering 3 permutations with 1 in ratio 2015-10-16 13:07:29,228 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:07:29,228 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:07:29,228 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=255.0 M 2015-10-16 13:07:29,228 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5262b65b84ab44e980f2c1f9b79d0074, keycount=745820, bloomtype=ROW, size=113.3 M, encoding=NONE, seqNum=11201396 2015-10-16 13:07:29,228 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/294b1f8242bf4f74bdda9690f28aef4a, keycount=466240, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=11248024 2015-10-16 13:07:29,228 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/94288a9a9ed943649614a169205dae35, keycount=466360, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=11294664 2015-10-16 13:07:29,243 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:07:29,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 4770 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:29,536 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 244ms (threshold=20ms); ack: seqno: 4782 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1129234, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:29,600 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630502_1099518555132] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 440 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 682407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:29,605 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 4794 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1226370, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:29,675 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 19159 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 732916, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:29,690 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 4806 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1244587, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:29,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630502_1099518555132] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 650 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 51612950, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:29,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 19542 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 51620074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:29,795 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:07:29,795 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:07:29,795 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:07:29,795 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:07:29,795 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:07:29,983 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 285ms (threshold=20ms); ack: seqno: 4818 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188394, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:30,062 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 4831 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993296, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:30,167 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630502_1099518555132] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 1200 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:30,167 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 21451 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 398840, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:30,167 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:07:30,167 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:07:30,167 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:07:30,167 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:07:30,167 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:07:30,297 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 228ms (threshold=20ms); ack: seqno: 4843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4525370, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:30,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 4855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895027, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:30,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 210ms (threshold=20ms); ack: seqno: 4866 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1173051, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:30,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630502_1099518555132] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 1845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940941, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:30,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 24227 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 555074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:30,671 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:07:30,671 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:30,671 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:07:30,671 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:30,671 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:30,725 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 4877 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062315, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:30,928 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 193ms (threshold=20ms); ack: seqno: 4891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:31,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 4903 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 883450, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:31,112 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 4915 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:31,389 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 28302 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 798962, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:31,389 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:31,389 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:31,389 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:31,389 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:31,390 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:31,391 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 271ms (threshold=20ms); ack: seqno: 4926 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1042946, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:31,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 4939 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 848295, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:31,675 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 225ms (threshold=20ms); ack: seqno: 4950 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1049504, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:31,782 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 4956 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1161332, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:31,828 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 4962 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 880218, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:31,882 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 4968 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801904, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:31,933 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 4979 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 816084, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:31,985 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 4983 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 818744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:32,062 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 4996 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870359, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:32,141 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 5010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964583, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:32,198 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 32969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 514775, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:32,198 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:32,198 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630503_1099518555133] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 2276 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 793940, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:07:32,199 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:32,199 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:32,198 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:32,199 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:32,200 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 5022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 855232, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:32,414 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 214ms (threshold=20ms); ack: seqno: 5023 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1128857, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:32,508 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630503_1099518555133] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 2647 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 821162, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:07:32,508 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 34462 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 562565, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:32,508 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:32,508 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:32,508 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:32,508 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:32,508 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:32,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 219ms (threshold=20ms); ack: seqno: 5096 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1181495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:32,701 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 5109 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 890139, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:32,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 35434 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 586621, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:32,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 5113 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835408, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:32,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630503_1099518555133] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 2928 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 768652, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:07:32,742 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:32,742 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:32,742 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:32,742 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:32,742 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:32,755 DEBUG [PriorityRpcServer.handler=3,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:07:32,755 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 13:07:32,755 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:07:32,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 5115 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975153, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:32,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 5126 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054737, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:32,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 5134 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941261, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:33,053 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 5153 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066773, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:33,154 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 5169 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:33,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 37891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 411625, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:33,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630503_1099518555133] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 3613 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 708393, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:07:33,248 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:33,248 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:07:33,248 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:33,257 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 5184 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:33,353 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 5196 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1213385, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:33,468 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 5208 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868228, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:33,477 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:07:33,613 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 5253 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 851801, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:33,624 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#56 average throughput is 56.64 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 13:07:33,643 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/eb857aa86e2a468d8e30306ee28f7beb as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eb857aa86e2a468d8e30306ee28f7beb 2015-10-16 13:07:33,651 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:07:33,655 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:07:33,655 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=160754256 > lower bound=16777216 2015-10-16 13:07:33,655 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=153.26 MB 2015-10-16 13:07:33,663 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5262b65b84ab44e980f2c1f9b79d0074, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5262b65b84ab44e980f2c1f9b79d0074 2015-10-16 13:07:33,671 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/294b1f8242bf4f74bdda9690f28aef4a, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/294b1f8242bf4f74bdda9690f28aef4a 2015-10-16 13:07:33,672 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 184ms (threshold=20ms) 2015-10-16 13:07:33,679 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Slow sync cost: 191 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:33,681 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630499_1099518555129] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 40117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 569548, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:33,688 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/94288a9a9ed943649614a169205dae35, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/94288a9a9ed943649614a169205dae35 2015-10-16 13:07:33,688 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 3 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into eb857aa86e2a468d8e30306ee28f7beb(size=255.0 M), total size for store is 16.8 G. This selection was in queue for 0sec, and took 4sec to execute. 2015-10-16 13:07:33,688 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=3, fileSize=255.0 M, priority=-1, time=36903958082131364; duration=4sec 2015-10-16 13:07:33,688 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:07:33,688 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 7 compacting, 2 eligible, 10 blocking 2015-10-16 13:07:33,689 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:07:33,689 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:07:33,689 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:07:33,696 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 5282 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2602884, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:33,697 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026044870 with entries=79416, filesize=124.49 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026053477 2015-10-16 13:07:33,700 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026036621 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026036621 2015-10-16 13:07:33,702 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630504_1099518555134] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1095043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:33,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630504_1099518555134] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 413843, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:33,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630505_1099518555135] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 53 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1738923, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:07:33,795 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 5326 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 9766954, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:33,795 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 13:07:33,795 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:07:33,795 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:07:33,795 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 13:07:33,795 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 13:07:34,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 5413 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859752, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:34,149 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 5423 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986791, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:34,292 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 5447 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917186, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:34,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 5459 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 970015, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:34,491 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630505_1099518555135] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 1116 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 660842, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:07:34,491 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 5482 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 703725, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:34,493 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630504_1099518555134] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 3339 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895568, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:34,567 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 5483 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 856656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:34,597 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 5488 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 771620, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:34,671 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11350491, memsize=153.3 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5981d834def6401986484d19a2222343 2015-10-16 13:07:34,676 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5981d834def6401986484d19a2222343 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5981d834def6401986484d19a2222343 2015-10-16 13:07:34,682 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5981d834def6401986484d19a2222343, entries=558220, sequenceid=11350491, filesize=84.9 M 2015-10-16 13:07:34,684 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~153.32 MB/160762480, currentsize=24.12 MB/25291760 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 1029ms, sequenceid=11350491, compaction requested=true 2015-10-16 13:07:34,684 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:07:34,684 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 7 compacting, 3 eligible, 10 blocking 2015-10-16 13:07:34,685 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 13:07:34,686 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:07:34,686 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:07:34,710 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 5494 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 780014, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:34,761 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 5507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 875999, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:34,822 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 5513 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921210, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:34,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 5538 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857534, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:35,085 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 5550 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915850, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:35,239 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630504_1099518555134] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 7519 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 506314, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:35,239 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:35,239 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:35,239 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:35,239 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 35ms (threshold=20ms) 2015-10-16 13:07:35,239 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:07:35,288 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 5687 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 799234, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:35,402 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 5696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:35,514 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 5707 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910691, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:35,628 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 5719 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 889855, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:35,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 5732 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995288, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:35,839 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 5744 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:35,885 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 5749 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910955, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:36,013 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 5774 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1109589, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:36,115 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 5856 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751736, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:36,115 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630504_1099518555134] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 12359 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489653, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:36,115 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:36,115 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:36,115 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:36,115 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:36,115 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:36,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630504_1099518555134] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 12520 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 64824900, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:36,212 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 65ms (threshold=20ms) 2015-10-16 13:07:36,212 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 65ms (threshold=20ms) 2015-10-16 13:07:36,212 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 64ms (threshold=20ms) 2015-10-16 13:07:36,212 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 65ms (threshold=20ms) 2015-10-16 13:07:36,212 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 65ms (threshold=20ms) 2015-10-16 13:07:36,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 169ms (threshold=20ms); ack: seqno: 5877 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 878999, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:36,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 275ms (threshold=20ms); ack: seqno: 5889 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1152119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:36,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 5900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 879380, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:37,012 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 312ms (threshold=20ms); ack: seqno: 5913 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102993, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:37,144 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630504_1099518555134] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 17709 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 387552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:37,144 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:37,145 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:37,145 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:37,145 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:37,146 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 5925 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1302269, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:37,395 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 242ms (threshold=20ms); ack: seqno: 5936 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1192944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:37,511 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 5948 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 916617, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:37,823 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 280ms (threshold=20ms); ack: seqno: 5961 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 916888, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:37,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 5973 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908637, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:38,098 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:38,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630504_1099518555134] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 23047 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 379657, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:38,098 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:07:38,098 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:07:38,098 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:07:38,098 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:07:38,178 DEBUG [PriorityRpcServer.handler=9,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:07:38,178 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134269872 > lower bound=16777216 2015-10-16 13:07:38,178 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:07:38,227 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 254ms (threshold=20ms); ack: seqno: 5985 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1192013, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:38,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 5997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926967, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:38,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 301ms (threshold=20ms); ack: seqno: 6009 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 837839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:38,862 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630506_1099518555136] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 990 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 653555, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:07:38,862 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.0, tune compaction throughput to 20.00 MB/sec 2015-10-16 13:07:38,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 190ms (threshold=20ms); ack: seqno: 6014 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1400448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:38,885 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630504_1099518555134] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 26548 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672137, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:38,963 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11397119, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8e9769bd1e8647c6ae3c55bc5b60f2ca 2015-10-16 13:07:38,968 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/8e9769bd1e8647c6ae3c55bc5b60f2ca as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8e9769bd1e8647c6ae3c55bc5b60f2ca 2015-10-16 13:07:38,991 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 6022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895399, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:38,991 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8e9769bd1e8647c6ae3c55bc5b60f2ca, entries=466240, sequenceid=11397119, filesize=70.8 M 2015-10-16 13:07:38,996 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134273200, currentsize=19.90 MB/20861920 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 818ms, sequenceid=11397119, compaction requested=true 2015-10-16 13:07:38,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:07:38,997 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 7 compacting, 4 eligible, 10 blocking 2015-10-16 13:07:38,998 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 876310754 starting at candidate #0 after considering 3 permutations with 1 in ratio 2015-10-16 13:07:38,998 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:07:38,998 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:07:38,999 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=835.7 M 2015-10-16 13:07:38,999 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4d6d4c12178e4c909876d90556b9c9c1, keycount=2797740, bloomtype=ROW, size=425.0 M, encoding=NONE, seqNum=11126809 2015-10-16 13:07:38,999 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eb857aa86e2a468d8e30306ee28f7beb, keycount=1678420, bloomtype=ROW, size=255.0 M, encoding=NONE, seqNum=11294664 2015-10-16 13:07:38,999 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5981d834def6401986484d19a2222343, keycount=558220, bloomtype=ROW, size=84.9 M, encoding=NONE, seqNum=11350491 2015-10-16 13:07:38,999 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8e9769bd1e8647c6ae3c55bc5b60f2ca, keycount=466240, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=11397119 2015-10-16 13:07:39,015 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:07:39,067 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 6034 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792821, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:39,121 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 6041 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 897646, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:39,204 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 6051 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794172, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:39,268 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 6056 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 780967, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:39,364 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 6075 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 829162, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:39,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 6092 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 715497, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:39,479 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 6099 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751144, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:39,609 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630504_1099518555134] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 30617 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 480496, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:39,609 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:39,609 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:39,609 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:39,755 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 267ms (threshold=20ms); ack: seqno: 6111 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982949, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:39,813 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 6117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1002302, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:39,933 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630498_1099518555128] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 6154 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 923198, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:07:40,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 883ms (threshold=20ms); ack: seqno: 165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012752, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:40,117 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 6300 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:40,293 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630504_1099518555134] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 34449 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 433570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:07:40,293 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:07:40,293 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:07:40,293 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:07:40,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 797ms (threshold=20ms); ack: seqno: 6358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1000622, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:41,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 874ms (threshold=20ms); ack: seqno: 332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1119528, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:41,027 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:07:41,063 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026053477 with entries=77861, filesize=122.05 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026061027 2015-10-16 13:07:41,065 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026044870 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026044870 2015-10-16 13:07:41,103 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630509_1099518555139] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971166, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:41,103 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 386 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1822079, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:41,103 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:41,103 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:41,103 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 6484 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089103, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:41,103 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:41,103 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:41,973 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630509_1099518555139] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 4939 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 632007, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:41,973 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:41,973 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:41,973 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:41,973 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:41,973 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:41,974 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 840ms (threshold=20ms); ack: seqno: 6525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1402966, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:41,998 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 6531 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:42,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 842ms (threshold=20ms); ack: seqno: 499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988174, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:42,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 6535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 952751, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:42,445 DEBUG [PriorityRpcServer.handler=7,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:07:42,445 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 13:07:42,445 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:07:42,700 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630509_1099518555139] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 9054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 14145008, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:42,839 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630509_1099518555139] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 9128 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 483141, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:42,840 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 127ms (threshold=20ms) 2015-10-16 13:07:42,840 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 127ms (threshold=20ms) 2015-10-16 13:07:42,840 INFO [sync.4] wal.FSHLog: Slow sync cost: 127 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:42,840 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 127ms (threshold=20ms) 2015-10-16 13:07:42,840 INFO [sync.3] wal.FSHLog: Slow sync cost: 127 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:42,840 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 127ms (threshold=20ms) 2015-10-16 13:07:42,840 INFO [sync.1] wal.FSHLog: Slow sync cost: 127 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:42,840 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 127ms (threshold=20ms) 2015-10-16 13:07:42,840 INFO [sync.0] wal.FSHLog: Slow sync cost: 127 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:42,840 INFO [sync.2] wal.FSHLog: Slow sync cost: 127 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:42,974 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 825ms (threshold=20ms); ack: seqno: 6694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1215835, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:43,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 885ms (threshold=20ms); ack: seqno: 666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020826, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:43,032 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 6703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188243, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:43,037 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630509_1099518555139] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 9520 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 607487, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:43,209 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 6853 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 894856, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:43,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630509_1099518555139] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 13355 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 452216, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:43,736 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:43,736 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:43,736 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:43,736 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:43,736 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:43,986 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#53 675819/5500620 (12.29%), rate=7001.40 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=20.00 MB/sec, activeCompactions=2] 2015-10-16 13:07:44,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 881ms (threshold=20ms); ack: seqno: 833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1247242, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:44,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 1034ms (threshold=20ms); ack: seqno: 6861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 275667507, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:44,312 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#53 sleep 660 ms because current throughput is 29.41 MB/sec, max allowed is 10.00 MB/sec, already slept 4 time(s) and total slept time is 3200 ms till now. 2015-10-16 13:07:44,545 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630509_1099518555139] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 17826 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 514975, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:44,546 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:44,546 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:44,546 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:44,546 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:44,546 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:44,722 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 691ms (threshold=20ms); ack: seqno: 846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 691460831, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:44,974 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 661ms (threshold=20ms); ack: seqno: 7028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1082766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:45,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 248ms (threshold=20ms); ack: seqno: 1000 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1142395, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:45,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 7032 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1230495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:45,193 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 7167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937041, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:45,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630509_1099518555139] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 21771 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 391912, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:45,304 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:45,304 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:45,304 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:45,304 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:45,304 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:45,974 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 762ms (threshold=20ms); ack: seqno: 7195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1150744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:46,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 871ms (threshold=20ms); ack: seqno: 1167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066947, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:46,124 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 1254 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 726052, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:46,124 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630509_1099518555139] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 26446 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 500206, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:46,124 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 7332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 629442, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:46,125 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:46,125 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:46,125 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:46,125 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:07:46,125 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:07:46,277 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 1318 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1173614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:47,014 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630509_1099518555139] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 31541 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 566992, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:47,014 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:47,014 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:47,014 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:47,014 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:47,014 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:47,016 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 868ms (threshold=20ms); ack: seqno: 7362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1115664, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:47,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 750ms (threshold=20ms); ack: seqno: 1334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 8580367, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:47,757 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630509_1099518555139] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 35496 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 70801254, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:47,757 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 13:07:47,757 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 13:07:47,757 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 13:07:47,757 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 13:07:47,788 INFO [sync.0] wal.FSHLog: Slow sync cost: 102 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:47,788 INFO [sync.2] wal.FSHLog: Slow sync cost: 102 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:47,757 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 71ms (threshold=20ms) 2015-10-16 13:07:47,788 INFO [sync.4] wal.FSHLog: Slow sync cost: 102 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:47,788 INFO [sync.3] wal.FSHLog: Slow sync cost: 102 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:47,788 INFO [sync.1] wal.FSHLog: Slow sync cost: 102 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:47,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630509_1099518555139] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 35501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 696200, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:47,866 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630509_1099518555139] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 35719 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 374267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:47,866 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:47,866 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:47,866 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:47,867 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:47,975 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 824ms (threshold=20ms); ack: seqno: 7529 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1210826, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:48,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 867ms (threshold=20ms); ack: seqno: 1501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1171479, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:48,026 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 7536 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1241534, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:48,115 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630509_1099518555139] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 36446 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 687267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:48,605 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:07:48,654 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026061027 with entries=77983, filesize=122.25 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026068605 2015-10-16 13:07:48,657 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630510_1099518555140] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1147567, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:48,768 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630510_1099518555140] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 419 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 547240, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:48,769 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:48,769 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:48,769 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:48,769 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:48,769 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:48,985 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 852ms (threshold=20ms); ack: seqno: 7696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 11384864, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:49,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 884ms (threshold=20ms); ack: seqno: 1668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1135640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:49,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 7703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1251832, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:49,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 7817 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1115852, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:49,271 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 7825 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794197, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:49,330 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 7831 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994338, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:49,394 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 7842 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1051743, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:49,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 7853 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 953568, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:49,488 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 7861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 901443, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:49,547 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630510_1099518555140] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 4664 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 411872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:49,547 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:49,547 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:49,547 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:49,547 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:49,547 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:07:49,975 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 487ms (threshold=20ms); ack: seqno: 7863 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1236370, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:50,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 873ms (threshold=20ms); ack: seqno: 1835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054938, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:50,045 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 7890 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794987, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:50,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 7905 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762491, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:50,236 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 7921 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911184, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:50,263 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 7928 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:50,366 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630510_1099518555140] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 9343 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 409853, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:50,367 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:07:50,367 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:07:50,367 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:07:50,367 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:07:50,367 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:07:50,381 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 7935 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963497, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:50,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 7949 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943783, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:50,636 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 7957 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1078416, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:50,763 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 7964 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 860888, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:51,009 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 216ms (threshold=20ms); ack: seqno: 7980 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1195380, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:51,024 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630507_1099518555137] hdfs.DFSClient: Slow ReadProcessor read fields took 869ms (threshold=20ms); ack: seqno: 2002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1117211, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:07:51,178 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:07:51,178 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:07:51,178 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630510_1099518555140] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 14162 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 421511, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:51,178 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:07:51,180 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 160ms (threshold=20ms); ack: seqno: 7988 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1214479, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:51,287 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 7995 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 817791, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:51,400 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 8002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956384, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:51,516 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 8010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927319, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:51,636 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 8025 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081619, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:51,704 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 8030 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 882912, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:51,762 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 8033 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 935696, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:52,008 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 221ms (threshold=20ms); ack: seqno: 8049 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1077709, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:52,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630511_1099518555141] hdfs.DFSClient: Slow ReadProcessor read fields took 616ms (threshold=20ms); ack: seqno: 2171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1394178, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:07:52,075 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630510_1099518555140] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 19298 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 520032, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:52,075 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630511_1099518555141] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 2186 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917487, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:07:52,075 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:52,075 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:52,075 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:52,075 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:52,285 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 253ms (threshold=20ms); ack: seqno: 8065 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154993, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:52,313 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 8072 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:52,434 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 8185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 882748, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:52,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 195ms (threshold=20ms); ack: seqno: 8199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066816, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:52,691 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 8204 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979512, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:52,817 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 8304 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 634331, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:52,817 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630510_1099518555140] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 23252 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 514721, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:52,818 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:07:52,818 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:07:52,818 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:07:52,818 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:07:52,818 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:07:52,874 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630508_1099518555138] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 8311 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1027374, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:07:53,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630511_1099518555141] hdfs.DFSClient: Slow ReadProcessor read fields took 838ms (threshold=20ms); ack: seqno: 2340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1294488, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:07:53,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 680ms (threshold=20ms); ack: seqno: 8369 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1436677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:07:53,701 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 8400 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989247, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:07:54,024 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630511_1099518555141] hdfs.DFSClient: Slow ReadProcessor read fields took 880ms (threshold=20ms); ack: seqno: 2507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1358004, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:07:54,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630510_1099518555140] hdfs.DFSClient: Slow ReadProcessor read fields took 520ms (threshold=20ms); ack: seqno: 27849 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 519648021, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:54,152 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 519ms (threshold=20ms) 2015-10-16 13:07:54,152 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 520ms (threshold=20ms) 2015-10-16 13:07:54,152 INFO [sync.1] wal.FSHLog: Slow sync cost: 519 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:54,152 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 521ms (threshold=20ms) 2015-10-16 13:07:54,152 INFO [sync.0] wal.FSHLog: Slow sync cost: 520 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:54,152 INFO [sync.4] wal.FSHLog: Slow sync cost: 520 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:54,152 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 520ms (threshold=20ms) 2015-10-16 13:07:54,152 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 520ms (threshold=20ms) 2015-10-16 13:07:54,152 INFO [sync.3] wal.FSHLog: Slow sync cost: 520 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:54,152 INFO [sync.2] wal.FSHLog: Slow sync cost: 520 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:54,574 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630510_1099518555140] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 29509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797600, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:54,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 1157ms (threshold=20ms); ack: seqno: 8535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 319486115, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:07:55,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630510_1099518555140] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 31843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 669229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:55,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 8563 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1604363, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:07:55,003 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:55,003 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:55,003 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:55,003 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:55,003 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:07:55,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630511_1099518555141] hdfs.DFSClient: Slow ReadProcessor read fields took 897ms (threshold=20ms); ack: seqno: 2674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1685340, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:07:55,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 574ms (threshold=20ms); ack: seqno: 8702 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1057173, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:07:55,719 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 8707 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019411, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:07:55,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 8737 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 742670, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:07:55,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630510_1099518555140] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 36200 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 448007, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:07:55,781 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:55,781 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:55,781 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:55,781 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:55,781 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:07:56,024 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630511_1099518555141] hdfs.DFSClient: Slow ReadProcessor read fields took 886ms (threshold=20ms); ack: seqno: 2843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1542678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:07:56,640 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 747ms (threshold=20ms); ack: seqno: 8871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1639082, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:07:56,684 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 8877 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1073825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:07:57,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630511_1099518555141] hdfs.DFSClient: Slow ReadProcessor read fields took 879ms (threshold=20ms); ack: seqno: 3010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1804764, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:07:57,138 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630511_1099518555141] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 3111 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 867551, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:07:57,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 854ms (threshold=20ms); ack: seqno: 9038 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1663607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:07:57,844 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 9191 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924056, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:07:58,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630511_1099518555141] hdfs.DFSClient: Slow ReadProcessor read fields took 841ms (threshold=20ms); ack: seqno: 3177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1490607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:07:58,640 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 774ms (threshold=20ms); ack: seqno: 9205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1472474, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:07:59,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630511_1099518555141] hdfs.DFSClient: Slow ReadProcessor read fields took 883ms (threshold=20ms); ack: seqno: 3344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1627707, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:07:59,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 877ms (threshold=20ms); ack: seqno: 9372 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1622602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:07:59,757 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 9379 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1479633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:00,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630511_1099518555141] hdfs.DFSClient: Slow ReadProcessor read fields took 883ms (threshold=20ms); ack: seqno: 3511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1305762, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:08:00,640 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 772ms (threshold=20ms); ack: seqno: 9539 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1372475, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:01,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630511_1099518555141] hdfs.DFSClient: Slow ReadProcessor read fields took 892ms (threshold=20ms); ack: seqno: 3678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7987255, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:08:01,121 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630511_1099518555141] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 3777 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 788967, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:08:01,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 881ms (threshold=20ms); ack: seqno: 9706 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1367587, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:01,680 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 9716 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924501, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:01,851 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 137ms (threshold=20ms); ack: seqno: 9740 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1116457, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:01,904 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 9764 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1110070, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:02,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630511_1099518555141] hdfs.DFSClient: Slow ReadProcessor read fields took 861ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1621648, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:08:02,040 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 9775 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050251, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:02,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 9810 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945355, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:02,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 170ms (threshold=20ms); ack: seqno: 9823 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174069, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:02,551 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 9848 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012987, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:02,677 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 9859 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1036257, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:02,893 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 10001 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1009656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:03,026 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630511_1099518555141] hdfs.DFSClient: Slow ReadProcessor read fields took 893ms (threshold=20ms); ack: seqno: 4012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1631490, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:08:03,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 10025 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1052061, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:03,138 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 10031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847141, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:03,687 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 544ms (threshold=20ms); ack: seqno: 10039 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1548134, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:03,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 10047 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102605, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:03,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 10049 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969720, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:03,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 10054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1100852, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:03,955 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 10067 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085467, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:04,024 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630513_1099518555143] hdfs.DFSClient: Slow ReadProcessor read fields took 877ms (threshold=20ms); ack: seqno: 4181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207101, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:08:04,071 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 10095 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070208, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:04,169 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 10114 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853958, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:04,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 10129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1036119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:04,298 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 10140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:04,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 10146 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 916779, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:04,492 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 10183 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1030054, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:04,606 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 10194 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:04,930 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 318ms (threshold=20ms); ack: seqno: 10206 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 18686109, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:05,024 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630513_1099518555143] hdfs.DFSClient: Slow ReadProcessor read fields took 881ms (threshold=20ms); ack: seqno: 4348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1232921, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:08:05,029 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 10218 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943613, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:05,198 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 163ms (threshold=20ms); ack: seqno: 10229 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1140852, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:05,273 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 10296 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 711281, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:05,366 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 10307 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 929065, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:05,493 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 10318 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014596, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:05,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 287ms (threshold=20ms); ack: seqno: 10332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1451285, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:05,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 10344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1099945, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:06,024 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630513_1099518555143] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 4517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1359310, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:08:06,179 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 259ms (threshold=20ms); ack: seqno: 10356 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1116934, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:06,325 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 10368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1216282, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:06,594 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 262ms (threshold=20ms); ack: seqno: 10381 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1453282, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:06,703 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 10393 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956162, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:06,986 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630512_1099518555142] hdfs.DFSClient: Slow ReadProcessor read fields took 277ms (threshold=20ms); ack: seqno: 10405 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1369620, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:08:07,024 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630513_1099518555143] hdfs.DFSClient: Slow ReadProcessor read fields took 899ms (threshold=20ms); ack: seqno: 4684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1150156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:08:08,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630513_1099518555143] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 4851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1349077, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:08:09,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630513_1099518555143] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 5018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006135, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:08:09,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 10509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1122485, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:09,370 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 152ms (threshold=20ms); ack: seqno: 10649 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1254061, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:09,493 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 10661 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 16096739, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:09,586 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 10672 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 977318, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:09,706 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 10690 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1107013, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:09,825 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 10703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979777, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:10,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630513_1099518555143] hdfs.DFSClient: Slow ReadProcessor read fields took 899ms (threshold=20ms); ack: seqno: 5185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1545174, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:08:10,136 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 307ms (threshold=20ms); ack: seqno: 10710 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1107882, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:10,254 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 10721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069689, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:10,371 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 10732 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1025153, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:10,414 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 10739 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:10,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 10756 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:10,488 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 10768 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 892581, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:10,625 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 10791 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1140359, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:10,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 10802 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915911, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:10,721 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 10821 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902681, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:10,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 10829 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 968403, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:10,849 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 10835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1473221, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:11,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 10860 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 968553, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:11,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630513_1099518555143] hdfs.DFSClient: Slow ReadProcessor read fields took 899ms (threshold=20ms); ack: seqno: 5352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1211495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:08:11,046 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 10873 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853256, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:11,137 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 10879 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936179, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:11,266 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 10892 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954250, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:11,309 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 10903 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926369, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:11,451 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 10915 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7202913, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:11,580 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 10927 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021827, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:11,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 10951 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1126112, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:11,777 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 10963 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835976, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:11,949 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 10994 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1096324, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:11,983 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 11005 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 840276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:12,026 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630513_1099518555143] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 5521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:08:12,127 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 138ms (threshold=20ms); ack: seqno: 11016 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984822, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:12,307 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 11040 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1033805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:12,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 11053 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1106511, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:12,544 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 11072 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 891097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:12,674 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 11098 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979576, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:12,821 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 11133 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1190041, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:13,006 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 176ms (threshold=20ms); ack: seqno: 11145 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127530, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:13,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630513_1099518555143] hdfs.DFSClient: Slow ReadProcessor read fields took 899ms (threshold=20ms); ack: seqno: 5688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1194422, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:08:13,104 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630513_1099518555143] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 5689 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 78323848, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:08:13,162 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 11180 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982679, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:13,311 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 11215 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1106886, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:13,346 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 11223 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1312996, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:13,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 11227 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 996079, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:13,682 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 238ms (threshold=20ms); ack: seqno: 11251 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1430888, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:13,716 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 11263 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1003063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:13,829 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 11271 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 970051, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:13,954 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 11286 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1124275, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:14,010 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 11287 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1132429, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:14,026 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630513_1099518555143] hdfs.DFSClient: Slow ReadProcessor read fields took 863ms (threshold=20ms); ack: seqno: 5855 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1512622, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:08:14,109 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 11317 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 961074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:14,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 234ms (threshold=20ms); ack: seqno: 11329 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1404100, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:14,410 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 11333 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1605148, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:14,501 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 11347 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 10141203, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:14,582 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 11366 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1063882, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:14,618 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 11385 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913627, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:14,676 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 11388 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1052503, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:14,767 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 11403 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188921, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:14,862 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 11426 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1133110, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:14,961 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 11436 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1109620, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:14,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 11443 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1001075, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:15,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630513_1099518555143] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 6022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1394209, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:08:15,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 11464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1145265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:15,072 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630513_1099518555143] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 6044 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 608591, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:08:15,102 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 11471 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 906539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:15,218 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 11482 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1031986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:15,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 11506 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967591, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:15,418 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 11530 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1301205, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:15,536 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 11541 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1177227, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:15,593 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 11549 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1027900, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:15,714 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 11555 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1226837, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:15,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 11562 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1039026, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:15,841 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 11567 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1252154, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:15,960 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 11585 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1203889, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:16,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630513_1099518555143] hdfs.DFSClient: Slow ReadProcessor read fields took 867ms (threshold=20ms); ack: seqno: 6189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1437930, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:08:16,106 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 11610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1052968, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:16,256 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 148ms (threshold=20ms); ack: seqno: 11614 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1248677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:16,584 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 322ms (threshold=20ms); ack: seqno: 11622 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1382771, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:16,773 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 11670 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010029, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:16,917 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 11682 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1065858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:16,979 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 11705 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904083, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:17,013 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 11718 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907487, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:17,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630515_1099518555145] hdfs.DFSClient: Slow ReadProcessor read fields took 896ms (threshold=20ms); ack: seqno: 6358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1637463, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:08:17,105 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 11730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985158, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:17,151 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 11742 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 878880, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:17,412 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 254ms (threshold=20ms); ack: seqno: 11753 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1498165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:17,771 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 353ms (threshold=20ms); ack: seqno: 11765 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1459487, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:17,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 11778 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1302919, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:18,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630515_1099518555145] hdfs.DFSClient: Slow ReadProcessor read fields took 896ms (threshold=20ms); ack: seqno: 6525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1222726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:08:18,077 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 11795 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038881, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:18,229 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 11808 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1150320, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:18,299 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 11814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1128496, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:18,694 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 374ms (threshold=20ms); ack: seqno: 11821 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1469398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:19,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630515_1099518555145] hdfs.DFSClient: Slow ReadProcessor read fields took 886ms (threshold=20ms); ack: seqno: 6694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1682587, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:08:19,071 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 373ms (threshold=20ms); ack: seqno: 11828 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1483765, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:19,125 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 11845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 931154, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:19,609 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 484ms (threshold=20ms); ack: seqno: 11846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 6560373, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:19,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 12005 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808099, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:19,889 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 148ms (threshold=20ms); ack: seqno: 12009 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1100603, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:19,968 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 12020 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:20,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630515_1099518555145] hdfs.DFSClient: Slow ReadProcessor read fields took 875ms (threshold=20ms); ack: seqno: 6861 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1406287, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:08:20,051 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 12032 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993613, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:20,270 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 212ms (threshold=20ms); ack: seqno: 12045 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1497408, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:20,549 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 277ms (threshold=20ms); ack: seqno: 12049 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1572126, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:20,593 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 12059 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1246325, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:20,804 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 205ms (threshold=20ms); ack: seqno: 12070 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1496622, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:20,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 12081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1437580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:21,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630515_1099518555145] hdfs.DFSClient: Slow ReadProcessor read fields took 894ms (threshold=20ms); ack: seqno: 7028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1657979, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:08:21,123 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 203ms (threshold=20ms); ack: seqno: 12093 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1322961, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:21,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 12106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014409, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:21,319 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 12119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1058709, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:21,513 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 187ms (threshold=20ms); ack: seqno: 12131 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1061351, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:21,597 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 12142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1107913, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:21,683 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 12154 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1181775, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:21,810 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 12280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 906702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:21,904 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 12299 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1046043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:21,994 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 12312 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081565, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:22,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630515_1099518555145] hdfs.DFSClient: Slow ReadProcessor read fields took 896ms (threshold=20ms); ack: seqno: 7195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1373574, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:08:22,080 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 12324 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 987618, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:22,297 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 209ms (threshold=20ms); ack: seqno: 12336 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1503002, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:22,373 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 12349 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111860, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:22,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 12360 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1118470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:22,676 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 211ms (threshold=20ms); ack: seqno: 12372 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1609464, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:22,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630514_1099518555144] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 12383 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1024694, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:23,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630515_1099518555145] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 7362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1468470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:08:23,093 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 236ms (threshold=20ms); ack: seqno: 12509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4808633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:23,182 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 12520 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1686995, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:23,275 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 12532 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1374452, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:23,532 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 251ms (threshold=20ms); ack: seqno: 12545 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1451148, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:23,724 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 183ms (threshold=20ms); ack: seqno: 12554 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1499091, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:23,859 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 12562 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1250019, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:23,956 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 12575 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1318935, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:24,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630515_1099518555145] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 7529 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1228938, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:08:24,044 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 12587 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1112682, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:24,108 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630515_1099518555145] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 7610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 711116, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:08:24,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 251ms (threshold=20ms); ack: seqno: 12600 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1595088, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:24,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 12612 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1469538, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:24,672 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 254ms (threshold=20ms); ack: seqno: 12623 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1550171, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:24,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 12635 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1441883, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:25,027 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 222ms (threshold=20ms); ack: seqno: 12646 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1517041, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:25,029 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630515_1099518555145] hdfs.DFSClient: Slow ReadProcessor read fields took 869ms (threshold=20ms); ack: seqno: 7696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1380975, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:08:25,137 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 12658 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1281479, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:25,414 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 272ms (threshold=20ms); ack: seqno: 12668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1577056, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:25,541 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 12681 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1395787, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:25,759 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 212ms (threshold=20ms); ack: seqno: 12693 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1376929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:25,837 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 12705 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1423352, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:26,029 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630515_1099518555145] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 7863 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1217341, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:08:26,155 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 312ms (threshold=20ms); ack: seqno: 12717 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1421814, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:26,199 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 12729 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1223123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:26,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 254ms (threshold=20ms); ack: seqno: 12741 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1516038, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:26,552 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 12752 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1595122, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:26,592 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 12756 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1275209, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:26,805 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 207ms (threshold=20ms); ack: seqno: 12764 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1513203, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:26,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 12777 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1545811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:26,974 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 12789 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1099438, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:27,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630515_1099518555145] hdfs.DFSClient: Slow ReadProcessor read fields took 901ms (threshold=20ms); ack: seqno: 8030 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2514046, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:08:27,211 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 228ms (threshold=20ms); ack: seqno: 12802 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1431337, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:27,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 12814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1370841, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:27,371 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 12821 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:27,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 12834 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 900670, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:27,651 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 238ms (threshold=20ms); ack: seqno: 12846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1333554, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:27,769 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 12858 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1275004, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:28,004 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 232ms (threshold=20ms); ack: seqno: 12866 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1445541, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:28,029 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630515_1099518555145] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 8199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1400982, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:08:28,064 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 12878 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 931723, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:28,375 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 232ms (threshold=20ms); ack: seqno: 12997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1447710, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:28,467 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 13009 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:28,556 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 13021 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969176, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:28,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 228ms (threshold=20ms); ack: seqno: 13033 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1309306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:28,882 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 13044 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1257019, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:29,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630517_1099518555147] hdfs.DFSClient: Slow ReadProcessor read fields took 856ms (threshold=20ms); ack: seqno: 8368 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1756561, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:29,069 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 183ms (threshold=20ms); ack: seqno: 13054 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956745, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:29,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630517_1099518555147] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 8486 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 732728, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:29,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 13058 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1071950, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:29,262 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 13105 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1114228, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:29,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 13118 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1146188, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:29,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 13129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982847, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:29,672 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 221ms (threshold=20ms); ack: seqno: 13141 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1233145, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:29,761 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 13153 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1015797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:29,852 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 13164 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1171112, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:29,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 13172 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 856731, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:30,029 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 13188 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1049042, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:30,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630517_1099518555147] hdfs.DFSClient: Slow ReadProcessor read fields took 866ms (threshold=20ms); ack: seqno: 8535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1112376, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:30,117 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 13200 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948166, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:30,214 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 13211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 955926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:30,307 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 13224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1135212, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:30,400 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 13236 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019949, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:30,591 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 13353 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1093023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:30,621 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 13359 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:30,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 160ms (threshold=20ms); ack: seqno: 13365 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1156726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:30,818 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 13376 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1112217, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:30,946 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 13387 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1077774, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:31,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630517_1099518555147] hdfs.DFSClient: Slow ReadProcessor read fields took 898ms (threshold=20ms); ack: seqno: 8702 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1411980, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:31,221 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 274ms (threshold=20ms); ack: seqno: 13389 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1285410, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:31,346 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 13400 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1103651, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:31,460 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 13412 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1268910, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:31,569 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 13423 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1203815, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:31,714 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 13447 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1284551, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:31,748 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 13459 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943015, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:31,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 13482 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1162124, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:32,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630517_1099518555147] hdfs.DFSClient: Slow ReadProcessor read fields took 897ms (threshold=20ms); ack: seqno: 8871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1486034, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:32,047 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 160ms (threshold=20ms); ack: seqno: 13494 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790937, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:32,174 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 13507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1139407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:32,335 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 140ms (threshold=20ms); ack: seqno: 13527 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1099293, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:32,487 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 13551 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1178290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:32,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 13564 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1042235, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:32,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 13586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1139908, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:32,889 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 13605 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1128506, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:32,917 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 13613 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043297, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:33,032 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630517_1099518555147] hdfs.DFSClient: Slow ReadProcessor read fields took 895ms (threshold=20ms); ack: seqno: 9038 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1189680, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:33,093 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 147ms (threshold=20ms); ack: seqno: 13635 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915729, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:33,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 13636 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920472, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:33,247 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 13659 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1137719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:33,441 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 187ms (threshold=20ms); ack: seqno: 13670 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1173293, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:33,612 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 163ms (threshold=20ms); ack: seqno: 13682 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1098387, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:33,669 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 13693 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1039923, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:33,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 13706 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062894, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:33,814 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 13711 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1049825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:33,949 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 13723 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196843, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:34,032 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630517_1099518555147] hdfs.DFSClient: Slow ReadProcessor read fields took 884ms (threshold=20ms); ack: seqno: 9205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1455222, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:34,101 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 13737 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904177, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:34,253 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 145ms (threshold=20ms); ack: seqno: 13748 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043964, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:34,388 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 13760 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021453, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:34,551 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 13785 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1142252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:34,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 13797 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1185326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:34,837 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 13808 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1223229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:34,888 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 13820 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050446, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:34,973 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 13826 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1148483, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:35,032 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630517_1099518555147] hdfs.DFSClient: Slow ReadProcessor read fields took 890ms (threshold=20ms); ack: seqno: 9372 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1530653, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:35,101 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 13838 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011918, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:35,315 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 13999 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:36,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630517_1099518555147] hdfs.DFSClient: Slow ReadProcessor read fields took 888ms (threshold=20ms); ack: seqno: 9539 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1410832, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:36,141 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 787ms (threshold=20ms); ack: seqno: 14059 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:36,243 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 14071 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1098018, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:36,344 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 14081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972639, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:36,485 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 14117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038836, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:36,598 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 14129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1051375, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:36,705 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 14140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1137254, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:36,818 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 14153 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1056588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:36,937 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 14165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1119264, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:37,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630517_1099518555147] hdfs.DFSClient: Slow ReadProcessor read fields took 891ms (threshold=20ms); ack: seqno: 9706 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1586829, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:37,044 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 14177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984898, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:37,154 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 14189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 872701, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:37,267 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 14201 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1039190, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:37,432 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 14238 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1027059, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:37,560 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 14257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1146386, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:37,770 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 14319 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1097434, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:37,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 138ms (threshold=20ms); ack: seqno: 14330 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1314846, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:38,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630517_1099518555147] hdfs.DFSClient: Slow ReadProcessor read fields took 891ms (threshold=20ms); ack: seqno: 9873 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1598076, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:38,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630517_1099518555147] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 9880 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2391654, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:38,067 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 147ms (threshold=20ms); ack: seqno: 14342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 826721, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:38,124 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 14347 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966787, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:38,221 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 14361 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:38,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 192ms (threshold=20ms); ack: seqno: 14367 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1198915, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:38,493 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 14378 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1065945, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:38,557 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 14390 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1182672, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:38,600 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 14408 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 935212, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:38,657 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 14416 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1169350, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:38,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 14449 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1112070, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 13:08:38,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 14476 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1074355, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:38,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 14488 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965789, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:38,975 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 14494 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023981, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:39,029 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#57 7489714/5500620 (136.16%), rate=10238.74 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=unlimited, activeCompactions=2] 2015-10-16 13:08:39,044 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630517_1099518555147] hdfs.DFSClient: Slow ReadProcessor read fields took 875ms (threshold=20ms); ack: seqno: 10040 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 13894894, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:39,073 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 14509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1179306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:39,109 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 14517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1128652, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:39,197 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 14520 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1074194, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:39,294 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 14536 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1061697, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:39,346 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 14542 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207074, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:39,392 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 14547 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1118645, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:39,499 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 14563 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1283566, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:39,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 14571 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988322, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:39,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630516_1099518555146] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 14578 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1086436, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:08:39,711 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 14598 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1393678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:39,821 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 14614 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1285089, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:39,930 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 14626 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1181303, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:40,053 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 14638 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1293389, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:40,183 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 14652 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1280810, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:40,227 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630517_1099518555147] hdfs.DFSClient: Slow ReadProcessor read fields took 1182ms (threshold=20ms); ack: seqno: 10041 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2968110, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:40,243 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 14668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 891962, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:40,348 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 14707 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945627, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:40,498 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 14744 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1200459, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:40,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 14768 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1160659, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:40,674 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630519_1099518555149] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 10802 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 763662, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:08:40,674 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 14773 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 970599, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:40,770 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 14779 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085205, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:40,898 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 14792 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1179697, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:41,016 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 14803 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1143488, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:41,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 14816 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196320, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:41,298 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 14841 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1191472, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:41,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 137ms (threshold=20ms); ack: seqno: 14852 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1244175, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:41,481 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 14862 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020589, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:41,481 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630519_1099518555149] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 12073 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758172, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:08:41,637 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 14900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1146381, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:41,752 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 14911 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1152091, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:41,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 14935 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1147367, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:41,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 14956 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 781990, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:42,052 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 14961 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909415, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:42,247 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630520_1099518555150] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 13280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 697427, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:08:42,412 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 281ms (threshold=20ms); ack: seqno: 15016 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1298212, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:42,439 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#57 average throughput is 12.81 MB/sec, slept 60 time(s) and total slept time is 52011 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 13:08:42,457 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c10e9a74381840e98d2e2bb782a18e3c as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c10e9a74381840e98d2e2bb782a18e3c 2015-10-16 13:08:42,472 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630510_1099518555140] hdfs.DFSClient: Slow ReadProcessor read fields took 16156ms (threshold=20ms); ack: seqno: 39339 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 768491, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:42,472 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:08:42,475 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:08:42,483 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4d6d4c12178e4c909876d90556b9c9c1, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/4d6d4c12178e4c909876d90556b9c9c1 2015-10-16 13:08:42,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 15028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1491722, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:42,492 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eb857aa86e2a468d8e30306ee28f7beb, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/eb857aa86e2a468d8e30306ee28f7beb 2015-10-16 13:08:42,500 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5981d834def6401986484d19a2222343, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5981d834def6401986484d19a2222343 2015-10-16 13:08:42,509 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8e9769bd1e8647c6ae3c55bc5b60f2ca, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/8e9769bd1e8647c6ae3c55bc5b60f2ca 2015-10-16 13:08:42,509 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into c10e9a74381840e98d2e2bb782a18e3c(size=835.5 M), total size for store is 16.9 G. This selection was in queue for 0sec, and took 1mins, 3sec to execute. 2015-10-16 13:08:42,509 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=835.7 M, priority=-1, time=36903967852482304; duration=1mins, 3sec 2015-10-16 13:08:42,509 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:08:42,509 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 7 compacting, 1 eligible, 10 blocking 2015-10-16 13:08:42,510 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:08:42,510 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:08:42,510 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:08:42,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 15041 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1263161, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:42,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 15052 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023641, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:42,778 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536922096 > lower bound=16777216 2015-10-16 13:08:42,778 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 13:08:42,876 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630510_1099518555140] hdfs.DFSClient: Slow ReadProcessor read fields took 403ms (threshold=20ms); ack: seqno: 39340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 854312, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:42,946 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 254ms (threshold=20ms); ack: seqno: 15063 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1450957, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:43,047 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 15076 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1086755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:43,292 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 238ms (threshold=20ms); ack: seqno: 15087 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1413379, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:43,393 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 15098 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1307293, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:43,609 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 209ms (threshold=20ms); ack: seqno: 15110 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1394166, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:43,723 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 15123 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1232667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:43,950 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 222ms (threshold=20ms); ack: seqno: 15134 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1307232, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:44,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630521_1099518555151] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 1933 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 734290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:08:44,034 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#53 9184888/9137108 (100.52%), rate=8435.52 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=unlimited, activeCompactions=1] 2015-10-16 13:08:44,038 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 15146 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2436654, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:44,288 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 245ms (threshold=20ms); ack: seqno: 15158 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1287099, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:44,348 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 15171 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1135648, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:44,450 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 15184 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992663, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:44,695 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 238ms (threshold=20ms); ack: seqno: 15195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1604341, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:45,804 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 1110ms (threshold=20ms); ack: seqno: 15196 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2333710, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:45,804 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630522_1099518555152] hdfs.DFSClient: Slow ReadProcessor read fields took 1178ms (threshold=20ms); ack: seqno: 2999 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1611676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:45,917 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 15285 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:45,996 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 15299 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 978946, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:46,240 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 237ms (threshold=20ms); ack: seqno: 15311 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1408617, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:46,333 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 15323 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 884923, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:46,606 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 265ms (threshold=20ms); ack: seqno: 15335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1247032, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:46,606 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630523_1099518555153] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 4416 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930682, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:08:46,696 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 15345 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1357811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:46,725 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11583560, memsize=512.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5189727ab45448b8ae80650ecbc4b00d 2015-10-16 13:08:46,731 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/5189727ab45448b8ae80650ecbc4b00d as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5189727ab45448b8ae80650ecbc4b00d 2015-10-16 13:08:46,736 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5189727ab45448b8ae80650ecbc4b00d, entries=1864360, sequenceid=11583560, filesize=283.2 M 2015-10-16 13:08:46,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630510_1099518555140] hdfs.DFSClient: Slow ReadProcessor read fields took 3863ms (threshold=20ms); ack: seqno: 39341 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 796878, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:08:46,738 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536921680, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 3960ms, sequenceid=11583560, compaction requested=false 2015-10-16 13:08:46,758 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 15370 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861049, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:47,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 245ms (threshold=20ms); ack: seqno: 15382 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1599383, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:47,080 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 15394 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 923737, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:47,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 15406 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1233988, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:47,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 240ms (threshold=20ms); ack: seqno: 15418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1652067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:47,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 15431 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1092992, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:47,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 146ms (threshold=20ms); ack: seqno: 15443 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1223869, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:47,843 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 15559 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037454, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:47,933 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 15571 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012313, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:48,190 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 232ms (threshold=20ms); ack: seqno: 15585 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1644849, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:48,281 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 15596 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920494, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:48,529 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 241ms (threshold=20ms); ack: seqno: 15607 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1564200, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:48,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 15618 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1396099, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:48,855 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 215ms (threshold=20ms); ack: seqno: 15629 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1566908, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:49,059 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 199ms (threshold=20ms); ack: seqno: 15638 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1448452, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:49,311 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 248ms (threshold=20ms); ack: seqno: 15646 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1543804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:49,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 15652 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1654202, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:49,619 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 203ms (threshold=20ms); ack: seqno: 15660 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1380212, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:49,703 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 15694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5691832, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:50,204 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 15953 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984261, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:50,257 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 15966 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954434, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:50,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 15975 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917892, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:50,312 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 15982 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940959, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:50,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 15990 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:50,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 16008 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934242, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:50,499 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 16032 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940453, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:50,536 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 16039 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902759, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:50,589 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 16060 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 860423, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:50,707 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 16099 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 978000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:50,770 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 16114 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026249, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:50,884 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 16122 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941915, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:50,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 16129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954065, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:50,950 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 16140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1001206, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,014 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 16159 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914019, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,059 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 16170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967554, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,094 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 16177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021341, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 16192 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 970343, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,200 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 16200 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1074441, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,227 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 16207 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993762, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,254 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 16217 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 897626, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,415 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 16268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1079217, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,452 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 16276 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 939629, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,479 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 16282 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966890, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,514 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 16295 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964144, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,541 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 16303 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,613 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 16323 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928925, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 16330 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979617, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,666 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 16338 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993149, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 16361 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928149, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 16369 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1016389, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,801 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 16376 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1003391, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,829 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 16384 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1078617, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,874 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 16390 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1032407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,925 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 16409 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971577, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,952 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 16416 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912136, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:51,989 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 16429 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971679, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:52,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 16440 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:52,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 16474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 978254, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:52,255 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 16490 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 960958, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:52,411 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 16556 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969287, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:52,515 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 16564 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017001, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:52,732 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 214ms (threshold=20ms); ack: seqno: 16570 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1570656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:52,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 16575 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1057837, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:52,886 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 16582 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1110610, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:53,125 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 236ms (threshold=20ms); ack: seqno: 16587 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1593350, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:53,190 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 16596 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1386142, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:53,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 16600 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021651, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:53,529 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 235ms (threshold=20ms); ack: seqno: 16606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1518270, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:53,620 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 16610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1646300, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:53,883 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 259ms (threshold=20ms); ack: seqno: 16618 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1554233, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:54,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 16624 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1547681, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:54,267 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 240ms (threshold=20ms); ack: seqno: 16629 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1628806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:54,377 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 16636 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1671744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:54,477 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 16640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1346731, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:54,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 264ms (threshold=20ms); ack: seqno: 16647 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7817804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:54,856 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 16655 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1412559, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:55,092 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 232ms (threshold=20ms); ack: seqno: 16662 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1498305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:55,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 16667 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1277786, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:55,396 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630518_1099518555148] hdfs.DFSClient: Slow ReadProcessor read fields took 223ms (threshold=20ms); ack: seqno: 16670 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 977602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:08:55,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 301ms (threshold=20ms); ack: seqno: 16747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1470539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:55,883 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 16753 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1063253, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:55,922 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 16758 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915750, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:56,042 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 16765 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1083146, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:56,065 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 16766 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794533, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:56,184 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 16770 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005415, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:56,352 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 142ms (threshold=20ms); ack: seqno: 16781 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 997138, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:56,503 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 147ms (threshold=20ms); ack: seqno: 16788 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1024192, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:56,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 16793 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 891426, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:56,646 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 16801 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945661, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:56,801 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 151ms (threshold=20ms); ack: seqno: 16806 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988738, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:56,969 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 145ms (threshold=20ms); ack: seqno: 16817 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1000226, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:57,108 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 16824 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1171446, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:57,140 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 16829 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930278, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:57,278 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 16836 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101419, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:57,309 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 16840 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:57,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 16847 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 968710, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:57,478 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 16851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1189482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:57,513 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 16859 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911542, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:57,632 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 16866 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1031893, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:57,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 16872 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1165580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:57,940 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 16995 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 959704, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:58,060 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 17003 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1104330, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:58,175 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 17010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 960181, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:58,289 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 17015 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1099108, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:58,409 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 17023 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 998883, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:58,521 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 17028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1040217, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:58,638 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 17036 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1220050, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:58,759 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 17043 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188278, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:58,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 17050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1093188, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:59,008 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 17057 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1163491, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:59,132 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 17062 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1168881, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:59,261 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 17069 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1200969, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:59,418 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 151ms (threshold=20ms); ack: seqno: 17074 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1241550, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:59,453 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 17081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1242731, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:59,571 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 17086 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1168597, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:59,614 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 17093 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:59,760 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 17108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1338173, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:08:59,904 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 141ms (threshold=20ms); ack: seqno: 17115 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1234752, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:00,039 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 17122 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1205413, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:00,073 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 17129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1000910, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:00,108 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 17138 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:00,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 17145 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921352, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:00,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 17151 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1167808, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:00,389 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 17156 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010104, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:00,510 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 17163 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 10448797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:00,532 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 17164 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1064439, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:00,817 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 284ms (threshold=20ms); ack: seqno: 17168 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1540607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:00,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 17175 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1331683, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:01,240 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 300ms (threshold=20ms); ack: seqno: 17181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1684911, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:01,332 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 17199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:01,452 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 17206 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127455, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:01,562 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 17211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060895, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:01,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 17217 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 981933, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:01,709 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 17223 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1191854, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:01,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 17225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908889, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:01,841 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 17229 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1056039, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:01,959 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 17237 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985425, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:02,103 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 17244 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1161790, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:02,137 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 17252 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:02,274 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 17264 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1099378, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:02,384 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 17270 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1141284, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:02,518 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 17275 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1128515, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:02,657 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 17281 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1136865, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:02,817 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 17293 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1041381, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:02,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 17298 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127312, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:03,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 134ms (threshold=20ms); ack: seqno: 17306 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1112215, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:03,110 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 17311 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 996073, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:03,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 151ms (threshold=20ms); ack: seqno: 17322 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1225494, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:03,424 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 17330 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1061597, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:03,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 17336 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 996151, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:03,652 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 17341 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1574301, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:03,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 17347 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1167436, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:03,892 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 17352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:04,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 17358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1097302, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:04,149 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 17364 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1114795, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:04,175 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 17371 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941859, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:04,295 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 17378 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062096, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:04,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 17385 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1118497, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:04,460 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 17391 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1083979, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:04,490 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 17396 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047752, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:04,596 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630510_1099518555140] hdfs.DFSClient: Slow ReadProcessor read fields took 17858ms (threshold=20ms); ack: seqno: 39342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 823992, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:04,620 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 17397 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1248221, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:04,734 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:09:04,774 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 25ms (threshold=20ms) 2015-10-16 13:09:04,791 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026068605 with entries=77818, filesize=121.99 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026144734 2015-10-16 13:09:04,794 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026053477 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026053477 2015-10-16 13:09:04,799 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026061027 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026061027 2015-10-16 13:09:04,833 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 17516 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088331, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:04,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 17522 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994042, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:05,084 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 17536 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1009465, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:05,193 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 17544 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924375, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:05,308 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 17551 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1219353, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:05,330 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 17552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956987, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:05,432 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 17558 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1015662, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:05,488 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630525_1099518555155] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1562 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 520520, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:05,488 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 17598 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 810934, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:05,488 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 30ms (threshold=20ms) 2015-10-16 13:09:05,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 17629 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983513, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:05,609 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 17640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 901789, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:05,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 17660 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 916022, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:05,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 17682 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1237169, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:05,774 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 17691 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2089630, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:05,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 17722 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920284, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:05,973 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 17752 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937993, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,005 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 17757 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993698, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 17765 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868824, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,094 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 17776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 860347, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,333 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 17868 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014349, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,388 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 17884 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912924, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,449 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 17899 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887720, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 17914 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975172, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,547 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 17921 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 877876, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,582 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 17930 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1123173, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,617 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 17937 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 953926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,652 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 17945 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927731, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,746 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 17970 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940091, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,785 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 17975 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 883609, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,820 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 17983 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1016067, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,931 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 18026 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 929354, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:06,975 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 18034 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885699, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,091 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 18065 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982508, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,134 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 18072 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,186 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 18080 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912594, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,230 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 18087 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,265 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18094 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928408, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 18102 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 840719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,353 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 18114 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 925702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,454 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18149 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995941, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,506 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 18163 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1003665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,551 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 18170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 881168, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,586 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18178 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936168, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 18190 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921506, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,690 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 18197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1041546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,736 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18214 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 817353, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 18240 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874150, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,870 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18249 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 883452, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,913 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 18256 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 840633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:07,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18272 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 771444, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 18280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836946, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,092 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18306 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 849432, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,149 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 18316 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857045, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 18351 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 18385 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830325, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,434 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 18403 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019557, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 18423 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839357, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,536 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18435 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 889382, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,537 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630525_1099518555155] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 5008 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 789152, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:08,570 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 18445 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885859, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,612 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 18450 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1119915, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 18466 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1053184, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,702 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 18474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013114, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,762 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 18482 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1179352, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,807 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 18498 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994210, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,842 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 18506 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964763, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,878 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18514 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037198, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:08,968 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18540 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066631, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,056 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 18560 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013386, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,091 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18567 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930145, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,170 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 18581 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964867, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,205 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,240 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18596 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 952783, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 18604 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948541, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,337 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 18622 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921852, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,382 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18630 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 878959, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18638 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950041, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,466 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 18658 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865278, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,528 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 18677 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870054, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,591 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 18700 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864373, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,645 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 18715 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 18723 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 929170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 18741 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 900679, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630524_1099518555154] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 18751 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956815, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:09,870 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 18787 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1142925, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:09,933 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 18802 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 793471, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:10,097 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 18858 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 961506, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:10,143 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 18877 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 851523, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:10,180 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 18893 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 803199, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:10,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 18900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 772419, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:10,326 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 18938 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 749454, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:10,473 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 18989 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947219, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:10,501 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 18997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741147, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:10,566 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 19020 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839368, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:10,611 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 19034 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808516, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:10,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 19057 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834012, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:10,687 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 19064 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861954, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:10,923 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 19195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:10,977 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 19212 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 881412, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:11,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 19223 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 818184, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:11,103 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 19246 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:11,158 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 19265 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 786660, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:11,211 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 19268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 723800, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:11,213 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630525_1099518555155] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 8117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 604485, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:11,236 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 19274 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782418, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:11,289 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 19288 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801610, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:11,395 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 19333 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 824569, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:11,422 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 19340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 745003, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:11,450 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 19347 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 745644, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:11,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 19365 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:11,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 19390 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 756555, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:11,823 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 19464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 768457, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:11,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630525_1099518555155] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 12599 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 428792, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:11,957 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:11,957 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:11,957 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:11,957 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:11,957 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:11,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 19531 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942643, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:11,986 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 19539 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 720582, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:12,041 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 19554 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5889937, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:12,150 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 19585 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 714038, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:12,627 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 295ms (threshold=20ms); ack: seqno: 19708 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1039712, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:12,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630525_1099518555155] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 17302 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 366474, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:12,739 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:12,739 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:12,739 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:12,739 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:12,739 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:12,741 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 19719 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1162523, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:13,082 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 333ms (threshold=20ms); ack: seqno: 19732 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 991769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:13,182 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 19743 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1029727, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:13,485 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 293ms (threshold=20ms); ack: seqno: 19755 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:13,630 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630525_1099518555155] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 22657 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 506417, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:13,630 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:13,630 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:13,630 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:13,630 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:13,631 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:13,632 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 19767 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1119792, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:13,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630525_1099518555155] hdfs.DFSClient: Slow ReadProcessor read fields took 147ms (threshold=20ms); ack: seqno: 22790 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 829328, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:13,939 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 299ms (threshold=20ms); ack: seqno: 19780 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2587483, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:14,043 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 19792 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1179373, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:14,333 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 282ms (threshold=20ms); ack: seqno: 19805 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1113211, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:14,461 DEBUG [PriorityRpcServer.handler=6,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:09:14,461 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134266096 > lower bound=16777216 2015-10-16 13:09:14,462 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:09:14,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630525_1099518555155] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 26733 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 575834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:14,616 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630527_1099518555157] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 112 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2653697, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:14,616 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630525_1099518555155] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 27190 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 708159, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:14,616 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:09:14,616 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:09:14,616 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:09:14,616 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:09:14,713 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 273ms (threshold=20ms); ack: seqno: 19957 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 978055, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:14,905 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 187ms (threshold=20ms); ack: seqno: 19964 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 11082409, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:15,337 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630525_1099518555155] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 30660 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 522947, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:15,337 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11630189, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1cab8de30c14490eadfade842e35c3b7 2015-10-16 13:09:15,337 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:09:15,337 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:09:15,337 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:09:15,337 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:09:15,337 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:09:15,342 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1cab8de30c14490eadfade842e35c3b7 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1cab8de30c14490eadfade842e35c3b7 2015-10-16 13:09:15,349 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 444ms (threshold=20ms); ack: seqno: 19965 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1114669, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:15,351 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1cab8de30c14490eadfade842e35c3b7, entries=466250, sequenceid=11630189, filesize=70.8 M 2015-10-16 13:09:15,353 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134275760, currentsize=22.01 MB/23077200 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 892ms, sequenceid=11630189, compaction requested=true 2015-10-16 13:09:15,353 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:09:15,353 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 7 compacting, 3 eligible, 10 blocking 2015-10-16 13:09:15,355 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 13:09:15,355 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:09:15,355 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:09:15,494 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 19998 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946723, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:15,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 20014 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017985, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:15,655 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 20021 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 831287, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:15,767 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 20050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917332, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:16,088 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 280ms (threshold=20ms); ack: seqno: 20092 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1084995, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:16,130 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 20099 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1094199, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:16,196 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630525_1099518555155] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 35614 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 723295, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:16,197 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:16,197 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:16,197 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:16,197 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:16,197 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:16,205 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 20104 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1018031, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:16,521 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 307ms (threshold=20ms); ack: seqno: 20117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127260, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:16,646 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 20128 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1120476, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:16,965 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 297ms (threshold=20ms); ack: seqno: 20141 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048298, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:17,092 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 20163 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 970134, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:17,139 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630525_1099518555155] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 40950 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 579633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:17,139 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:17,139 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:17,139 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:09:17,139 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:17,379 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 279ms (threshold=20ms); ack: seqno: 20175 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1212148, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:17,415 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:09:17,448 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026144734 with entries=77843, filesize=122.03 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026157415 2015-10-16 13:09:17,449 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026068605 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026068605 2015-10-16 13:09:17,488 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 20187 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1082451, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:17,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 268ms (threshold=20ms); ack: seqno: 20199 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1401409, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:17,908 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 20211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1262720, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:18,073 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630528_1099518555158] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 3583 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 393181, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:18,073 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:18,073 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:18,073 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:18,073 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:18,074 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:09:18,151 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 236ms (threshold=20ms); ack: seqno: 20223 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1192961, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:18,198 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 20227 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1198214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:18,320 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 20235 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1191021, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:18,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 285ms (threshold=20ms); ack: seqno: 20260 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1204791, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:18,649 DEBUG [PriorityRpcServer.handler=5,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:09:18,649 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134276400 > lower bound=16777216 2015-10-16 13:09:18,649 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.00 MB 2015-10-16 13:09:18,675 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630528_1099518555158] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 7487 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 651099, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:18,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 20271 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1356330, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:18,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630529_1099518555159] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 237 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 694077, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:18,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630528_1099518555158] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 8378 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 434792, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:18,873 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:18,873 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:18,873 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:18,874 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:19,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 276ms (threshold=20ms); ack: seqno: 20283 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1369726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:19,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 20296 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937610, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:19,189 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630528_1099518555158] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 9391 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 727046, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:19,424 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 277ms (threshold=20ms); ack: seqno: 20320 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1526152, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:19,448 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11676819, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/44378bf72e9147e4b8e8577fd8045ba3 2015-10-16 13:09:19,453 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/44378bf72e9147e4b8e8577fd8045ba3 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/44378bf72e9147e4b8e8577fd8045ba3 2015-10-16 13:09:19,459 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/44378bf72e9147e4b8e8577fd8045ba3, entries=466260, sequenceid=11676819, filesize=70.8 M 2015-10-16 13:09:19,460 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134279440, currentsize=19.13 MB/20060880 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 811ms, sequenceid=11676819, compaction requested=true 2015-10-16 13:09:19,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:09:19,460 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 7 compacting, 4 eligible, 10 blocking 2015-10-16 13:09:19,462 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio 2015-10-16 13:09:19,462 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:09:19,462 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:09:19,525 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 20331 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1399604, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:19,612 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630528_1099518555158] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 11673 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 531106, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:19,612 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 20389 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 831971, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:19,612 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:19,612 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:19,612 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:19,612 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:19,612 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:19,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 20397 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1113805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:19,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 20534 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1073965, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:20,154 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 240ms (threshold=20ms); ack: seqno: 20546 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1269218, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:20,295 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 20550 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1295678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:20,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630528_1099518555158] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 16478 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 508900, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:20,405 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:09:20,406 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:09:20,406 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:09:20,406 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:09:20,406 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:09:20,655 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 356ms (threshold=20ms); ack: seqno: 20558 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1080548, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:20,711 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 20577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 906350, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:20,774 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 20581 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932736, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:20,955 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 175ms (threshold=20ms); ack: seqno: 20589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043089, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:21,197 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 220ms (threshold=20ms); ack: seqno: 20600 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1042514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:21,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630528_1099518555158] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 21795 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 515211, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:21,272 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:21,272 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:21,273 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:21,273 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:21,273 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:21,274 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 20617 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:21,334 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 20630 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 899295, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:21,396 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 20654 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830480, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:21,568 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 20684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839343, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:21,700 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 20712 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 882758, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:21,818 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 20725 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1032543, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:21,920 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 20737 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1099460, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:22,097 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630528_1099518555158] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 26871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 395480, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:22,097 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:09:22,097 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:09:22,097 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:09:22,097 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:09:22,099 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 171ms (threshold=20ms); ack: seqno: 20750 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:22,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 20767 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1076019, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:22,381 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 20779 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1051979, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:22,516 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 20791 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1032598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:22,681 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630526_1099518555156] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 20832 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942520, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:22,748 DEBUG [PriorityRpcServer.handler=13,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:09:22,748 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 13:09:22,748 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:09:22,748 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 7 compacting, 4 eligible, 10 blocking 2015-10-16 13:09:22,750 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio 2015-10-16 13:09:22,750 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:09:22,750 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:09:22,874 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 163ms (threshold=20ms); ack: seqno: 20848 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1159516, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:22,931 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630528_1099518555158] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 31894 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 673110, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:22,931 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 20860 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904532, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:22,931 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:22,931 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:09:22,931 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:22,931 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:22,931 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:09:23,061 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 20871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1084212, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:23,094 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 20879 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4265131, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:23,242 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 20912 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1029731, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:23,370 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 20935 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943353, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:23,560 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 20978 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044981, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:23,697 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 21002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088790, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:23,750 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630528_1099518555158] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 36838 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 432890, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:23,750 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:09:23,751 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:09:23,751 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:23,751 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:23,751 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:09:23,752 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 21014 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 977700, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:23,866 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 21027 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 899960, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:23,980 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630528_1099518555158] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 37512 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 770821, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:23,993 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 21050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830433, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:24,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 21062 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 973593, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:24,232 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 21085 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1217977, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:24,238 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:09:24,294 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630528_1099518555158] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 39286 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 15547034, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:24,294 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 28ms (threshold=20ms) 2015-10-16 13:09:24,298 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026157415 with entries=77857, filesize=122.05 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026164238 2015-10-16 13:09:24,301 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1086570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:24,302 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026144734 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026144734 2015-10-16 13:09:24,566 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 21436 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813076, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:24,566 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 1336 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 401704, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:24,566 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:24,566 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:24,566 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:24,566 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:24,567 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:25,116 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 4282 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 520267, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:25,116 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 22090 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 763996, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:25,116 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:25,116 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:25,116 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:25,116 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:09:25,452 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 22474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838310, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:25,517 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 22485 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1018547, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:25,716 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:09:25,716 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 7378 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 391471, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:25,716 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:25,716 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:25,716 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:25,716 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:25,814 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 7763 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 571534, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:25,921 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 266ms (threshold=20ms); ack: seqno: 22654 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1285811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:25,979 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 22744 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:26,077 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 22780 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:26,249 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 22914 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1002128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:26,341 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630530_1099518555160] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 22922 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1318137, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:09:26,497 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 11630 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 441959, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:26,497 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:26,497 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:26,497 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:26,497 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:27,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630532_1099518555162] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 23011 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 687550, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:09:27,099 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630532_1099518555162] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 23066 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 765350, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:09:27,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 16120 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 504929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:27,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630532_1099518555162] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 23257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 680248, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:09:27,304 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:27,304 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:27,304 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:27,304 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:27,871 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630532_1099518555162] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 23935 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 748153, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:09:27,871 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 18896 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 665326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:27,871 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:27,871 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:27,871 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:27,871 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:28,438 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630532_1099518555162] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 24608 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908374, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:09:28,438 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 21692 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 461442, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:28,438 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:28,438 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:28,438 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:28,438 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:28,438 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:28,756 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630532_1099518555162] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 25006 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 19987059, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:09:28,994 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630533_1099518555163] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 25281 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 860875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:28,994 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 24535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 559280, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:28,994 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:28,994 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:28,994 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:28,994 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:28,994 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:29,229 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 25186 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934488, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:29,603 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 26979 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 609156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:29,603 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630533_1099518555163] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 26035 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692444, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:29,603 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:29,603 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:29,603 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:29,603 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:29,603 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:30,163 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 29735 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 547214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:30,163 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630533_1099518555163] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 26704 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 722300, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:30,163 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:30,163 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:30,163 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:30,163 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:30,163 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:30,403 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630533_1099518555163] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 26976 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 771316, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:30,724 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630534_1099518555164] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 27334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 772628, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:09:30,724 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:30,724 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 32703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 518972, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:30,724 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:30,724 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:30,796 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630534_1099518555164] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 27383 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861238, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:09:31,296 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630534_1099518555164] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 27957 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 6154484, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:09:31,296 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:31,297 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 35701 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 606058, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:31,846 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:31,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 38580 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 460939, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:31,846 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:31,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630534_1099518555164] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 28624 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 676769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:09:31,846 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:31,846 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:31,846 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:31,960 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:09:32,020 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630531_1099518555161] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 39364 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 32194979, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:32,027 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:09:32,031 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026164238 with entries=77784, filesize=121.94 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026171960 2015-10-16 13:09:32,035 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1091853, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:09:32,126 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=809, evicted=0, evictedPerRun=0.0 2015-10-16 13:09:32,401 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 1884 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 565872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:09:32,401 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630536_1099518555166] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 29331 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 698007, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:32,401 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:32,401 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:32,401 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:32,401 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:32,401 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:32,940 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 4641 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 498060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:09:32,940 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630536_1099518555166] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 30007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 757061, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:32,940 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:32,940 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:32,940 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:32,940 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:32,941 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:09:33,504 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 7289 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 585110, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:09:33,504 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630536_1099518555166] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 30721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1136196, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:09:33,504 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:09:33,504 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:09:33,505 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:09:33,505 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:33,505 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:09:34,070 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630537_1099518555167] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 31363 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 674796, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:09:34,070 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 10253 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 495959, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:09:34,070 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:34,070 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:34,070 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:34,070 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:34,070 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:09:34,453 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 11640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 788226, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:09:34,688 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630537_1099518555167] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 32122 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 803101, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:09:34,688 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 12696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 510831, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:09:34,688 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:34,688 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:34,688 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:34,688 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:34,689 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:35,242 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:09:35,265 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 62ms (threshold=20ms) 2015-10-16 13:09:35,265 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630537_1099518555167] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 32815 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 660011, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:09:35,266 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 15414 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 720588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:09:35,690 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630537_1099518555167] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 33271 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943458, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:09:35,926 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 18874 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 405779, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:09:35,926 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:35,926 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:35,926 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:09:35,926 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:35,926 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:09:35,939 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630537_1099518555167] hdfs.DFSClient: Slow ReadProcessor read fields took 209ms (threshold=20ms); ack: seqno: 33326 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1215166, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:09:36,090 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 33442 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853258, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:36,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 33491 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 867376, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:36,249 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 33546 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 998239, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:36,347 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 33596 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1126873, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:36,448 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 33653 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1061872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:36,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 33704 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062508, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:36,630 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 22629 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 504744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:09:36,631 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:09:36,631 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:09:36,631 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:09:36,631 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:09:36,632 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 33756 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102653, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:36,718 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 33813 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1063514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:36,809 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 33866 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1203650, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:36,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 33920 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1285123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:37,004 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 33969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1202844, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:37,118 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 34022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1168127, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:37,218 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 34072 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1161438, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:37,316 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 34124 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1159407, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=894156, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 13:09:37,419 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 34174 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965138, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:37,545 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 34224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1159796, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:37,651 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 34277 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1245070, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:37,904 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 34620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066931, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:38,169 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 34980 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 740975, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:38,340 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630538_1099518555168] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 35210 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014790, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 13:09:39,150 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630539_1099518555169] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 36334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 799824, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:09:40,065 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630540_1099518555170] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 37698 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801449, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:40,910 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630540_1099518555170] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 38948 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 962139, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:41,005 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630540_1099518555170] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 39040 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:41,085 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630540_1099518555170] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 39115 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 707899, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:41,953 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630541_1099518555171] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 40404 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 878050, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:09:42,310 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630541_1099518555171] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 40916 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 768546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:09:43,385 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630542_1099518555172] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 41718 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2591689, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:09:44,034 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#53 20360218/9137108 (222.83%), rate=28172.77 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=unlimited, activeCompactions=1] 2015-10-16 13:09:44,324 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630542_1099518555172] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 43133 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 691205, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:09:45,220 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630543_1099518555173] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 44489 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782784, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:45,946 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630543_1099518555173] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 45522 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1033637, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:46,238 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630543_1099518555173] hdfs.DFSClient: Slow ReadProcessor read fields took 180ms (threshold=20ms); ack: seqno: 45700 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 997105, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:46,364 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630543_1099518555173] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 45837 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 791241, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:46,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630543_1099518555173] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 45846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054011, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:09:47,310 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630544_1099518555174] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 47197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 734035, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:09:48,269 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630545_1099518555175] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 48552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 643909, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:09:49,192 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630545_1099518555175] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 49906 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 681257, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:09:49,808 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630546_1099518555176] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 50843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 931108, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:50,140 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630546_1099518555176] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 51272 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 6013607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:50,806 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 52230 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985234, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:50,935 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 52398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 761187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:51,124 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 52632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 719484, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:51,813 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 272ms (threshold=20ms); ack: seqno: 53268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1371358, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:51,972 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 53430 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070389, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:52,143 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 53590 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 981740, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:52,196 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 53621 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 903634, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:52,261 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 53624 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1584119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:52,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 53772 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905208, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:52,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 53928 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1240910, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:52,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 54002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 719171, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:52,798 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 54010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1239299, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:52,880 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 54038 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1138078, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:52,946 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 54045 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1246964, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:53,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 54094 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066693, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:53,094 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 54101 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127722, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:53,161 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 54150 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1052396, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:53,260 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 54156 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154199, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:53,310 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 54167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060537, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:53,378 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630547_1099518555177] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 54172 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127171, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:09:53,441 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630548_1099518555178] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 54209 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1105393, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:09:53,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630548_1099518555178] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 54260 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013270, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:09:53,634 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630548_1099518555178] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 54311 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1166665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:09:53,716 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630548_1099518555178] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 54365 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4131380, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:09:53,802 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630548_1099518555178] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 54411 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 991144, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:09:53,891 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630548_1099518555178] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 54463 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1083907, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:09:53,979 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630548_1099518555178] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 54514 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988297, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:09:54,068 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630548_1099518555178] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 54561 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1184531, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:09:54,161 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630548_1099518555178] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 54611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928384, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:09:54,253 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630548_1099518555178] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 54660 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 882962, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:09:54,334 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630548_1099518555178] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 54715 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1113335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:09:54,903 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630548_1099518555178] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 55359 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 668109, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:09:56,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630549_1099518555179] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 56705 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 678734, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:57,087 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630549_1099518555179] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 57838 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965716, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:57,301 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630549_1099518555179] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 58081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 670996, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:09:57,700 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630550_1099518555180] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 58678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937543, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:09:58,236 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630550_1099518555180] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 59431 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 721467, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:09:59,150 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630551_1099518555181] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 60787 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 824703, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:00,083 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630551_1099518555181] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 62160 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 759082, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:01,041 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630552_1099518555182] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 63515 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 740489, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:10:01,917 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630552_1099518555182] hdfs.DFSClient: Slow ReadProcessor read fields took 265ms (threshold=20ms); ack: seqno: 64445 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 264298620, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:10:02,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630552_1099518555182] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 64534 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:10:02,288 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 178ms (threshold=20ms); ack: seqno: 64694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1249229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:02,452 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 64853 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1120304, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:02,497 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 64869 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 886437, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:02,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 64880 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1179984, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:02,725 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 65029 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1246310, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:03,004 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 65313 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921147, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:03,178 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 65444 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928374, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:03,349 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 65587 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1172701, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:03,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 65729 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 970260, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:03,637 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 65771 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044279, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:03,696 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 65777 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 991808, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:03,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 65826 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896513, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:03,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 65832 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005221, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:03,988 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 65879 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1106286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:04,055 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 65884 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 891586, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:04,142 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 65958 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924757, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:04,202 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 65963 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 981942, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:04,312 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 66046 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898208, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:04,411 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 66120 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919370, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:04,505 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 66173 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1145914, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:04,584 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 66226 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:04,619 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 66229 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 894055, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:04,690 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 66281 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 900305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:04,782 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 66332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 987695, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:04,872 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 66389 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:04,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 66441 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022239, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:05,089 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 66491 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 949683, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:05,209 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630553_1099518555183] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 66611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 783386, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:05,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630554_1099518555184] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 66783 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865017, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:10:05,927 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630554_1099518555184] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 67586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 643166, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:10:06,602 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630554_1099518555184] hdfs.DFSClient: Slow ReadProcessor read fields took 130ms (threshold=20ms); ack: seqno: 68388 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101256, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:10:07,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630555_1099518555185] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 68960 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874593, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:08,237 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630555_1099518555185] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 70321 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 681630, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:08,349 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630555_1099518555185] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 70406 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 918065, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:08,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630555_1099518555185] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 70546 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1061697, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:08,714 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630555_1099518555185] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 70694 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1086082, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:08,908 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630555_1099518555185] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 70849 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:09,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630556_1099518555186] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 70996 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1205271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:09,292 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630556_1099518555186] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 71141 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1099199, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:09,535 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630556_1099518555186] hdfs.DFSClient: Slow ReadProcessor read fields took 140ms (threshold=20ms); ack: seqno: 71293 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1059236, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:09,690 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630556_1099518555186] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 71435 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1001719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:09,855 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630556_1099518555186] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 71576 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844925, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:09,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630556_1099518555186] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 71689 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 717083, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:10,034 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630556_1099518555186] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 71696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 980196, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:10,415 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630556_1099518555186] hdfs.DFSClient: Slow ReadProcessor read fields took 163ms (threshold=20ms); ack: seqno: 71994 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011604, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:10,576 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630556_1099518555186] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 72158 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:10,746 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630556_1099518555186] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 72308 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 772946, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:10,920 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630556_1099518555186] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 72469 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1028558, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:11,095 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630556_1099518555186] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 72634 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:11,269 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630556_1099518555186] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 72795 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797634, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:11,369 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630556_1099518555186] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 72808 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967173, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:11,543 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 72968 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1129771, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:11,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 73044 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 671663, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:11,687 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 73051 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:11,785 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 73135 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930126, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:11,836 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 73140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895328, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:12,010 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 73297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1052358, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:12,184 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 73459 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 977504, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:12,266 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 73555 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 774804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:12,353 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 73563 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 976269, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:12,580 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 73712 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008824, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:12,727 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 73809 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984943, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:12,827 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 73835 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1024922, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:13,186 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 74277 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 894547, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:13,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 74314 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 806978, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:13,332 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 74394 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 696250, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:13,433 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 74402 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1002632, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:13,606 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 74562 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989243, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:13,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 180ms (threshold=20ms); ack: seqno: 74713 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043330, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:14,041 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 74874 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 939769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:14,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630557_1099518555187] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 74901 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 53679678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:14,373 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630558_1099518555188] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 75200 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874245, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:10:14,544 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630558_1099518555188] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 75366 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873698, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:10:14,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630558_1099518555188] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 75396 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 846168, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:10:14,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630558_1099518555188] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 75469 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1009125, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:10:14,950 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630558_1099518555188] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 75632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 958909, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:10:15,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630558_1099518555188] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 75745 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 593711, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:10:15,183 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630558_1099518555188] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 75760 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 884154, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:10:15,288 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630558_1099518555188] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 75800 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983255, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:10:15,476 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630558_1099518555188] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 75957 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1133797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:10:15,609 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630558_1099518555188] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 76027 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836411, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:10:15,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630558_1099518555188] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 76194 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 991761, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:10:16,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630558_1099518555188] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 76345 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 958215, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:10:16,192 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630558_1099518555188] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 76513 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936971, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:10:16,615 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630558_1099518555188] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 77091 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 740972, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:10:16,736 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 77172 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:16,907 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 77224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1166062, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:16,977 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 77276 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1028964, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:17,046 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 77327 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010592, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:17,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 77375 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154071, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:17,201 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 77424 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845771, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:17,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 77475 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017374, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:17,347 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 77525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037946, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:17,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 77580 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1035542, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:17,520 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 77629 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989840, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:17,691 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 167ms (threshold=20ms); ack: seqno: 77635 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1238087, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:17,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 77682 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2668498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:17,807 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 77687 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154090, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:17,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 77734 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 949036, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:17,909 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 77739 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972397, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:17,981 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 77788 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920248, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:18,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 77793 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1000550, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:18,056 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 77802 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1002775, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:18,113 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 77808 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1058605, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:18,207 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 77840 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1147119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:18,268 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 77845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992912, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:18,447 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 78002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1127607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:18,634 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 78139 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984704, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:18,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 78225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 962973, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:18,757 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 78232 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870799, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:18,935 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 78436 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 621738, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:19,158 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 78716 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751544, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:19,415 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 79028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992163, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:19,558 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 79183 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1945611, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:19,591 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630559_1099518555189] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 79190 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1215925, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:19,682 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630560_1099518555190] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 79339 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5285506, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:10:20,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630560_1099518555190] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 79791 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905347, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:10:20,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630560_1099518555190] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 79795 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945789, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:10:20,225 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630560_1099518555190] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 79803 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1124618, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:10:21,184 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630560_1099518555190] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 81159 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 747603, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:10:21,854 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630561_1099518555191] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 82146 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853328, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:22,134 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630561_1099518555191] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 82515 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 691260, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:22,629 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630561_1099518555191] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 83068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969086, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:22,981 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630561_1099518555191] hdfs.DFSClient: Slow ReadProcessor read fields took 348ms (threshold=20ms); ack: seqno: 83076 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1537959, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:23,316 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630561_1099518555191] hdfs.DFSClient: Slow ReadProcessor read fields took 332ms (threshold=20ms); ack: seqno: 83083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1508348, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:23,520 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630561_1099518555191] hdfs.DFSClient: Slow ReadProcessor read fields took 198ms (threshold=20ms); ack: seqno: 83093 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1714488, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:23,697 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630561_1099518555191] hdfs.DFSClient: Slow ReadProcessor read fields took 171ms (threshold=20ms); ack: seqno: 83106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1379900, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:23,949 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630561_1099518555191] hdfs.DFSClient: Slow ReadProcessor read fields took 244ms (threshold=20ms); ack: seqno: 83118 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1647750, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:24,220 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630561_1099518555191] hdfs.DFSClient: Slow ReadProcessor read fields took 250ms (threshold=20ms); ack: seqno: 83129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1401185, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:24,289 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630561_1099518555191] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 83141 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1273128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:24,449 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630561_1099518555191] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 83274 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946064, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:24,612 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 83423 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1031869, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:24,786 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 83570 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 962481, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:24,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 83721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085295, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:25,155 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 83878 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1065731, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:25,231 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 83881 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 882312, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:25,296 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 83891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 980006, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:25,476 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 84041 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 906629, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:25,656 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 84181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 996612, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:25,823 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 84252 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887689, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:26,024 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 84407 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 851333, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:26,243 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 84552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008232, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:26,421 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 84712 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1049204, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:26,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 137ms (threshold=20ms); ack: seqno: 84869 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 957225, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:26,824 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 85030 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994369, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:26,989 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 85189 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010137, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:27,070 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 85244 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 695691, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:27,147 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 85254 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1068546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:27,324 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630562_1099518555192] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 85417 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985687, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:27,502 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630563_1099518555193] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 85587 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1015908, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:27,665 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630563_1099518555193] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 85739 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979865, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:27,918 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630563_1099518555193] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 86051 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853078, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:27,962 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630563_1099518555193] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 86059 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 831338, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:28,354 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630563_1099518555193] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 86618 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 748669, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:28,962 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630563_1099518555193] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 87526 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005592, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:10:29,292 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630564_1099518555194] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 87981 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 769396, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:29,939 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630564_1099518555194] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 88908 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885808, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:30,292 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630564_1099518555194] hdfs.DFSClient: Slow ReadProcessor read fields took 244ms (threshold=20ms); ack: seqno: 89072 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1672958, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:30,441 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630564_1099518555194] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 89241 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 881114, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:30,535 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630564_1099518555194] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 89335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762080, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:30,596 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630564_1099518555194] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 89348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:30,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630564_1099518555194] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 89505 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1195643, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:30,996 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630564_1099518555194] hdfs.DFSClient: Slow ReadProcessor read fields took 163ms (threshold=20ms); ack: seqno: 89557 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1193518, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:31,041 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630564_1099518555194] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 89569 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:31,131 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 89636 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1067072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:31,183 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 89644 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1130360, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:31,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 89688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1057558, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:31,328 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 89698 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1228719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:31,396 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 89739 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1024546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:31,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 89747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1046235, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:31,541 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 89791 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1068561, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:31,617 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 89801 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911281, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:31,691 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 89841 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779233, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:31,774 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 89847 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3708827, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:31,918 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 89883 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1110420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:31,999 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 89923 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072264, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:32,092 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 89933 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:32,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 89973 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934523, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:32,269 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 89980 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863244, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:32,327 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 89984 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 876578, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:32,451 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 89991 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066122, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:32,491 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 89995 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940109, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:32,613 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 90002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1065630, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:32,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 90085 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1035338, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:33,226 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 90695 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 621168, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:33,401 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630565_1099518555195] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 90843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804561, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:34,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630566_1099518555196] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 92050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2605329, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:10:34,791 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630566_1099518555196] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 92641 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936510, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:10:34,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630566_1099518555196] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 92707 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994560, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:10:35,182 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630566_1099518555196] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 92900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 5212605, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:10:35,945 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630566_1099518555196] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 93424 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838196, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:10:36,216 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630566_1099518555196] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 93707 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 822578, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:10:36,677 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630567_1099518555197] hdfs.DFSClient: Slow ReadProcessor read fields took 141ms (threshold=20ms); ack: seqno: 93881 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1034761, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:36,840 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630567_1099518555197] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 94022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026114, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:36,999 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630567_1099518555197] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 94175 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804970, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:37,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630567_1099518555197] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 94322 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797050, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:37,346 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630567_1099518555197] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 94469 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:37,518 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630567_1099518555197] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 94612 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 893691, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:37,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630567_1099518555197] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 94761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995622, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:37,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630567_1099518555197] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 94785 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 703510, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:37,806 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630567_1099518555197] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 94791 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912136, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:37,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630567_1099518555197] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 94937 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 879916, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:38,172 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630567_1099518555197] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 95082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1084570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:38,321 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630567_1099518555197] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 95173 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047578, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 13:10:38,986 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630568_1099518555198] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 96149 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 699949, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:10:39,315 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630568_1099518555198] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 96591 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866476, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:10:39,993 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630568_1099518555198] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 97511 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 626649, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:10:40,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630568_1099518555198] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 97782 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1099447, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:10:40,452 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630568_1099518555198] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 97910 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1029173, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:10:40,633 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 98067 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 897414, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:40,856 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 98211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1133016, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:41,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 98354 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902768, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:41,390 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 98866 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 668033, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:41,867 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 155ms (threshold=20ms); ack: seqno: 99328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021425, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:41,950 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 99379 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895964, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:42,033 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 99433 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 955514, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:42,099 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 99482 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859496, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:42,190 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 99538 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983824, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:42,290 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 99590 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 999996, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:42,378 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 99646 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913410, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:42,471 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 99702 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917419, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:42,560 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 99752 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836492, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:42,649 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 99807 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992918, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:42,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 99857 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 996625, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:42,834 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 99908 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1003165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:42,930 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 99959 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069916, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:43,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630569_1099518555199] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 100010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1107644, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:43,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630570_1099518555200] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 100045 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1439578, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:43,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630570_1099518555200] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 100094 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1053866, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:43,327 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630570_1099518555200] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 100150 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044729, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:43,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630570_1099518555200] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 100200 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 906513, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:43,477 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630570_1099518555200] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 100223 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 713616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:43,539 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630570_1099518555200] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 100250 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 957562, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:43,663 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630570_1099518555200] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 100328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 7513576, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:43,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630570_1099518555200] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 100648 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982946, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:44,034 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#53 43844448/9137108 (479.85%), rate=59203.21 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=unlimited, activeCompactions=1] 2015-10-16 13:10:44,174 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630570_1099518555200] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 100915 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 105601478, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:44,579 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630570_1099518555200] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 101577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794807, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:44,669 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630570_1099518555200] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 101688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4941989, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:45,557 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630571_1099518555201] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 102942 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 719358, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:10:46,520 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630572_1099518555202] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 104299 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 663536, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:10:47,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630572_1099518555202] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 105606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 765120, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:10:47,487 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630572_1099518555202] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 105669 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 679363, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:10:48,449 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630573_1099518555203] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 107027 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 660880, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:10:49,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630574_1099518555204] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 108995 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1137160, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:50,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630574_1099518555204] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 109320 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207736, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:50,294 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630574_1099518555204] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 109486 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1310140, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:50,401 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630574_1099518555204] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 109524 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1065753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:50,573 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630574_1099518555204] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 109683 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1169443, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:50,652 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630574_1099518555204] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 109761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 629984, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:50,711 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630574_1099518555204] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 109770 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 846641, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:50,883 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630574_1099518555204] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 109906 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930336, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:51,063 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630574_1099518555204] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 110059 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066814, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:51,210 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630574_1099518555204] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 110155 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:10:52,136 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 110545 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863461, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:52,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 110659 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:52,520 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 110814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1035970, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:52,678 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 110976 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022017, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:52,821 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher: Waited 90073ms on a compaction to clean up 'too many store files'; waited long enough... proceeding with flush of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:10:52,821 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536921776 > lower bound=16777216 2015-10-16 13:10:52,821 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 13:10:52,842 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 157ms (threshold=20ms); ack: seqno: 110987 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154529, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:52,927 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 16215ms (threshold=20ms); ack: seqno: 23045 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:52,980 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 111029 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1033206, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 111036 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 929546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,061 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 111040 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974759, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,061 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630576_1099518555206] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 137 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1179916, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:10:53,095 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 111041 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1099488, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,144 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 111048 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1105350, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,202 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 111052 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 976681, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,293 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 111064 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013927, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 111075 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 962679, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,426 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 111080 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975181, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,465 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 111087 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836447, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,525 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 111092 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,589 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 111099 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1121826, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,635 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 111106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964578, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,700 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 111118 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913042, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 111122 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1034616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 111129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869759, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,838 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 111134 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946910, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,885 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 111142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1004567, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:53,980 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 111154 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969202, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:54,175 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 111311 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 781085, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:54,354 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 111409 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993301, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:54,440 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 111422 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984384, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:54,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 111584 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1058555, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:54,882 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 111724 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:55,036 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 111874 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 860734, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:55,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 112024 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729760, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:55,299 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630577_1099518555207] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 3718 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 676117, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:55,299 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 112106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692315, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:55,538 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630575_1099518555205] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 112414 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 26596383, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:10:55,892 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11863260, memsize=512.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/63aea3d2bfbc43aabed3ac0887732da7 2015-10-16 13:10:55,898 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/63aea3d2bfbc43aabed3ac0887732da7 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/63aea3d2bfbc43aabed3ac0887732da7 2015-10-16 13:10:55,903 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/63aea3d2bfbc43aabed3ac0887732da7, entries=1864370, sequenceid=11863260, filesize=283.2 M 2015-10-16 13:10:55,904 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 2977ms (threshold=20ms); ack: seqno: 23046 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 593959, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:10:55,904 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536921360, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 3083ms, sequenceid=11863260, compaction requested=true 2015-10-16 13:10:55,904 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:10:55,904 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 12 store files, 7 compacting, 5 eligible, 10 blocking 2015-10-16 13:10:55,907 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 5 files of size 1618639655 starting at candidate #0 after considering 6 permutations with 2 in ratio 2015-10-16 13:10:55,907 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:10:55,907 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:10:55,907 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=1.5 G 2015-10-16 13:10:55,907 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c10e9a74381840e98d2e2bb782a18e3c, keycount=5500620, bloomtype=ROW, size=835.5 M, encoding=NONE, seqNum=11397119 2015-10-16 13:10:55,907 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5189727ab45448b8ae80650ecbc4b00d, keycount=1864360, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=11583560 2015-10-16 13:10:55,907 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1cab8de30c14490eadfade842e35c3b7, keycount=466250, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=11630189 2015-10-16 13:10:55,907 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/44378bf72e9147e4b8e8577fd8045ba3, keycount=466260, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=11676819 2015-10-16 13:10:55,907 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/63aea3d2bfbc43aabed3ac0887732da7, keycount=1864370, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=11863260 2015-10-16 13:10:55,979 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630578_1099518555208] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 113045 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 731981, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:56,041 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:10:56,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630578_1099518555208] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 113140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 851528, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:56,174 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 70 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1308934, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:56,275 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630578_1099518555208] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 113215 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922254, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:56,376 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 232 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072873, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:56,476 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630578_1099518555208] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 113369 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1130048, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:56,524 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 387 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1049527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:56,544 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630578_1099518555208] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 113433 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842662, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:56,630 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 407 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1130951, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:56,740 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 429 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1370387, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:56,765 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630578_1099518555208] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 113731 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 799866, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:56,867 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 478 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1190158, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:56,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868013, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:56,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630578_1099518555208] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 113895 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 721529, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:57,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 512 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1260335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:57,125 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630578_1099518555208] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 114059 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 929434, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:57,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1053916, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:57,325 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630578_1099518555208] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 114200 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956984, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:10:57,426 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 858 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947840, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:57,535 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 879 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974728, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:57,651 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 901 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926371, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:57,755 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630581_1099518555211] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 114665 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1655309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:57,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630581_1099518555211] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 114676 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1500605, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:57,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 1075 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1049997, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:57,936 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 1120 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895266, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:58,051 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 1142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 783864, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:58,112 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 1164 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:58,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630581_1099518555211] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 115083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1385304, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:58,142 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 1168 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:58,243 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 1187 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922044, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:58,288 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630581_1099518555211] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 115228 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1117755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:58,389 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 1255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869269, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:58,422 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630581_1099518555211] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 115375 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1052521, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:58,524 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 1309 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 951431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:58,589 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630581_1099518555211] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 115517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1124040, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:58,691 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 1419 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930734, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:58,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 1435 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 846535, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:58,737 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630581_1099518555211] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 115670 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993927, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:58,833 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630581_1099518555211] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 115679 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885121, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:58,888 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630581_1099518555211] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 115691 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804881, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:58,989 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 1659 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992013, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:59,099 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 1680 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 991983, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:59,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 1702 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902544, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:59,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630581_1099518555211] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 116192 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 955424, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:59,456 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 1894 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966535, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:59,554 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630581_1099518555211] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 116344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:59,613 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630581_1099518555211] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 116384 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 752014, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:59,615 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630580_1099518555210] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 2063 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 939637, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:10:59,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630581_1099518555211] hdfs.DFSClient: Slow ReadProcessor read fields took 241ms (threshold=20ms); ack: seqno: 116391 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1550035, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:10:59,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 317ms (threshold=20ms); ack: seqno: 2089 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1169291, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:00,047 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 2111 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 957098, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:00,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 2145 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942336, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:00,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630581_1099518555211] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 116635 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:00,262 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 2204 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 890626, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:00,312 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630583_1099518555213] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 116792 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983157, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:00,345 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630583_1099518555213] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 116798 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 952356, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:00,353 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 2290 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 820759, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:00,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 2297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1132300, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:00,485 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630583_1099518555213] hdfs.DFSClient: Slow ReadProcessor read fields took 138ms (threshold=20ms); ack: seqno: 116803 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940413, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:00,531 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630583_1099518555213] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 116810 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898787, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:00,569 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 2450 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1826916, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:00,657 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630583_1099518555213] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 116857 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1084512, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:00,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630583_1099518555213] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 116864 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1087397, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:00,870 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 268ms (threshold=20ms); ack: seqno: 2496 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1336566, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:00,892 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630583_1099518555213] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 117018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1076815, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:00,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 2594 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888114, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:01,059 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630583_1099518555213] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 117197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 711038, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:01,061 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 2612 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947931, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:01,132 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630583_1099518555213] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 117203 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1002322, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:01,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 154ms (threshold=20ms); ack: seqno: 2635 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1121171, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:01,330 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630583_1099518555213] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 117354 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:01,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 2791 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819114, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:01,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 2804 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 877072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:01,549 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 2810 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:01,613 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 2825 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 837680, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:01,669 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 2832 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 843953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:01,768 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 2847 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 931029, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:01,797 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 2851 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833729, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:01,819 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630583_1099518555213] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 118007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811721, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:01,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 2871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921122, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:01,956 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 2877 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 951534, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:02,045 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630583_1099518555213] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 118266 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 609798, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:02,047 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 2892 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1036354, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:02,125 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 2918 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910594, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:02,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 2921 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1024380, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:02,232 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 2941 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:02,273 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 2948 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819583, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:02,341 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 2962 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827752, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:02,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 2967 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993473, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:02,494 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 2985 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954234, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:02,538 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 2990 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975836, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:02,622 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 3007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928387, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:02,718 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 3029 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069604, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:02,762 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 3035 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005874, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:02,785 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 119150 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1002236, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:02,863 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 119157 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 949607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:02,973 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 119170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983688, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:03,009 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 119177 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842829, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:03,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 3341 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008443, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:03,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 119188 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 999806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:03,293 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 3661 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005846, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:03,394 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 267ms (threshold=20ms); ack: seqno: 119202 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:03,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 3813 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1108750, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:03,473 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 119268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825144, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:03,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 119272 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023176, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:03,629 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 119279 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964566, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:03,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 119291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:03,839 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 119407 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 925208, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:03,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 165ms (threshold=20ms); ack: seqno: 4142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1040973, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:03,880 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 119423 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862269, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:03,922 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630582_1099518555212] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 4156 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1007847, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:03,933 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 119429 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 990695, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:04,016 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 119443 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943556, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:04,069 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 4271 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:04,192 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 4414 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 770090, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:04,194 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 175ms (threshold=20ms); ack: seqno: 119449 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1142523, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:04,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 4422 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1176214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:04,346 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 141ms (threshold=20ms); ack: seqno: 119456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1177822, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:04,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 119467 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911455, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:04,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 119471 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989208, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:04,491 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 4732 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 791064, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:04,501 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 119479 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1000030, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:04,666 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 4895 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2661627, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:04,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 276ms (threshold=20ms); ack: seqno: 119483 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1449978, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:04,797 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 5057 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 832214, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:04,898 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 119498 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1205357, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:05,068 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 9163ms (threshold=20ms); ack: seqno: 23047 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861846, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:05,070 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 163ms (threshold=20ms); ack: seqno: 119509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942208, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:05,138 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 5356 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825830, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:05,238 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 119581 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870192, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:05,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 23634 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1433683, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:05,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 119603 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 784039, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:05,284 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:11:05,284 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 32ms (threshold=20ms) 2015-10-16 13:11:05,284 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:11:05,284 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:11:05,284 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 31ms (threshold=20ms) 2015-10-16 13:11:05,285 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 5510 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896897, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:05,338 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 5518 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 855055, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:05,339 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 119609 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 916854, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:05,457 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 5547 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919557, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:05,573 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 5558 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943800, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:05,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 24799 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 464016, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:05,766 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:05,766 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:05,767 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 5720 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1182215, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:05,767 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 193ms (threshold=20ms); ack: seqno: 119907 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010447, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:05,820 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 5727 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902761, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:05,847 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 119914 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905070, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:05,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 5750 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006454, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:06,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 119990 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 871038, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:06,122 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 5922 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 770671, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:06,233 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 25575 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 583533, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:06,233 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 6024 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 856153, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:06,233 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:06,233 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:06,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 120141 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928600, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:06,299 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 6031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:06,322 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 120148 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927683, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:06,369 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 6072 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985068, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:06,483 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 120205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 962802, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:06,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 6208 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937414, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:06,786 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630585_1099518555215] hdfs.DFSClient: Slow ReadProcessor read fields took 177ms (threshold=20ms); ack: seqno: 6240 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:06,829 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 120499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089196, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:06,834 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 6252 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1638747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:06,955 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 120534 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 949180, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:07,130 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 120662 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1018428, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:07,143 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 187ms (threshold=20ms); ack: seqno: 6450 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1076100, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:07,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 120670 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 959635, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:07,271 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 6490 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888437, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:07,310 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 26998 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 634984, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:07,310 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 6501 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 806366, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:07,311 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:11:07,311 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:11:07,312 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 120801 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946006, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:07,369 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 120809 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:07,371 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 6509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833005, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:07,591 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 218ms (threshold=20ms); ack: seqno: 120816 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 11801878, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:07,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 158ms (threshold=20ms); ack: seqno: 6792 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1155138, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:07,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 6798 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 999183, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,045 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 242ms (threshold=20ms); ack: seqno: 6819 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1255984, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,102 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 6827 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1052639, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,155 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 6841 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013870, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,218 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 6846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1071372, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,286 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 6863 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1090076, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,345 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 6868 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070446, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,424 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 6884 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081450, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 6891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1107036, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,557 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 6905 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1228083, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,595 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 6909 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1159198, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,663 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 6927 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1027314, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,706 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 6932 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154126, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,747 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 6952 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207209, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,817 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 6958 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1016560, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,892 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 6974 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081040, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:08,914 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 1262ms (threshold=20ms); ack: seqno: 27392 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758576, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:08,914 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 1259ms (threshold=20ms) 2015-10-16 13:11:08,914 INFO [sync.4] wal.FSHLog: Slow sync cost: 1259 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:08,914 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 1262ms (threshold=20ms) 2015-10-16 13:11:08,914 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 1259ms (threshold=20ms) 2015-10-16 13:11:08,914 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 1263ms (threshold=20ms) 2015-10-16 13:11:08,914 INFO [sync.3] wal.FSHLog: Slow sync cost: 1261 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:08,914 INFO [sync.2] wal.FSHLog: Slow sync cost: 1262 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:08,914 INFO [sync.0] wal.FSHLog: Slow sync cost: 1258 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:08,914 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 1258ms (threshold=20ms) 2015-10-16 13:11:08,914 INFO [sync.1] wal.FSHLog: Slow sync cost: 1257 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:08,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630584_1099518555214] hdfs.DFSClient: Slow ReadProcessor read fields took 1344ms (threshold=20ms); ack: seqno: 120870 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1000681, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:08,976 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 6980 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830115, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:09,050 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 6998 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934279, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:09,120 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 7003 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842915, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:09,227 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 7014 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 944438, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:09,297 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 7031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 889156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:09,390 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 7121 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 780369, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:09,390 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630587_1099518555217] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 121418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863917, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:09,390 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 27849 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 588860, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:09,390 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:11:09,390 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:11:09,390 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:11:09,465 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 7154 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 849803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:09,591 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 7185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972980, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:09,693 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630587_1099518555217] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 121804 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 882229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:09,801 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 7348 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 881870, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:09,934 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 7416 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988509, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:10,042 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 7439 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1117789, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:10,105 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 7472 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 783875, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:10,105 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 28551 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 22339513, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:10,105 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630587_1099518555217] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 122308 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 723178, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:10,105 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:10,105 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:10,105 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:10,105 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:10,105 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:10,207 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 7578 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873102, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:10,614 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630587_1099518555217] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 122935 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088026, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:10,655 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630587_1099518555217] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 122950 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 754364, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:10,656 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 29106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701868, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:10,657 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 8109 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:10,762 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 8120 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:10,863 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 123108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1176378, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:10,962 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630586_1099518555216] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 8253 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812367, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:11,064 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 123264 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1177807, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:11,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 8404 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 977391, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:11,266 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 123423 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1231213, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:11,366 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 8527 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1334536, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:11,469 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 123575 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1233522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:11,523 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 123608 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888188, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:11,525 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 29969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988764, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:11,525 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 8675 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006616, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:11,629 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 123622 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1093589, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:11,726 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 8838 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 817754, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:11,828 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 123755 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196385, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:11,936 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 8977 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:11,976 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 8989 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1856484, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:12,076 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 143ms (threshold=20ms); ack: seqno: 123907 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154763, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:12,177 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 9138 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921041, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:12,278 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 124058 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1016985, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:12,389 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 124180 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 657007, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:12,390 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 31083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 676621, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:12,390 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 9289 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 951284, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:12,495 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 9300 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:12,596 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 124328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012841, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:12,696 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 9442 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1191094, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:12,797 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 124473 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896512, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:12,896 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 9575 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986561, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:12,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 33449 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1601969, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:12,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 9654 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 869750, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:12,992 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:11:12,992 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:11:12,992 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:11:12,992 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:11:12,992 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:11:12,994 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 124617 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1172556, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:13,098 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 9668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971870, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:13,160 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 124774 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 880276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:13,179 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 9740 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792065, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:13,293 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 124817 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 955245, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:13,394 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 9866 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887686, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:13,498 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 124966 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:13,506 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 9964 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 798045, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:13,547 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 36216 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 559983, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:13,547 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630588_1099518555218] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 124982 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 708729, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:13,547 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:13,547 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:13,547 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:13,547 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:13,547 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:13,548 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 9969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 791308, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:13,651 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 9976 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865660, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:13,704 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 125164 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:13,762 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 9983 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873967, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:13,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 10140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835022, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:14,044 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 125417 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982410, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:14,175 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 125550 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 688588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:14,176 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 159ms (threshold=20ms); ack: seqno: 38641 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 815476, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:14,177 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 217ms (threshold=20ms); ack: seqno: 10153 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1285235, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:14,224 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 125558 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 774309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:14,225 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 10160 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 890729, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:14,321 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 125590 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975096, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:14,367 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 125596 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779160, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:14,381 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 10237 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924999, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:14,542 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 125640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044918, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:14,561 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630589_1099518555219] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 10366 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853435, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:14,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630535_1099518555165] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 41910 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 412270, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:14,784 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:14,784 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:14,784 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:14,785 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:11:14,785 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:11:14,785 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 125766 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967568, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:14,863 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 125773 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963955, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:14,917 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 125800 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 868691, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:14,961 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:11:14,986 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 21ms (threshold=20ms) 2015-10-16 13:11:15,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 125848 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864336, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:15,008 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026171960 with entries=77869, filesize=122.07 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026274961 2015-10-16 13:11:15,009 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026157415 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026157415 2015-10-16 13:11:15,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630592_1099518555222] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 981671, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:11:15,016 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026164238 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026164238 2015-10-16 13:11:15,192 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 125955 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 867515, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:15,240 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 125979 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 849135, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:15,309 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 126003 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974941, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:15,409 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630592_1099518555222] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 1848 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 552039, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:11:15,409 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:15,409 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:15,409 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:15,409 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:15,409 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:15,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 126031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986693, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:15,452 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 10846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1001978, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:15,537 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 10876 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979955, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:15,545 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 126067 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913694, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:15,582 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 10883 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154576, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:15,590 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 126079 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904069, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:15,618 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 10890 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1041183, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:15,628 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 126092 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932504, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:15,646 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 10898 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1021535, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:15,691 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 10910 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089838, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:15,703 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 126114 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905010, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:15,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 126126 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909120, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:15,775 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 10933 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 959326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:15,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 126133 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917356, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:15,836 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 126142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865281, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:15,885 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 10970 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:15,915 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 10977 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965511, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:15,959 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 10985 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048859, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:15,996 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 10992 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1007856, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:16,014 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 126196 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787466, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,023 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 10999 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1075017, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:16,060 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 11007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:16,070 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 126212 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857098, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 126218 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 786395, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,167 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630592_1099518555222] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 5823 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 716116, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:11:16,167 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 11037 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1024667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:16,167 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:11:16,167 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:11:16,167 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:16,167 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:16,167 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:16,168 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 126225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050177, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,247 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 11044 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085125, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:16,260 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 126229 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870994, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,317 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 126248 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 791311, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 11102 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037933, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:16,410 DEBUG [PriorityRpcServer.handler=15,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:11:16,410 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 13:11:16,410 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:11:16,414 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 126272 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 710532, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,441 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 11127 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005528, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:16,481 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 126291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946520, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 11142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:16,518 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 11150 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965957, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:16,535 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 126309 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736008, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,546 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 11157 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 925867, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:16,572 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 126319 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 692530, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,627 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 126334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 711368, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,658 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 126338 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792743, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 11194 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919126, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:16,703 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 11201 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038368, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:16,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 126356 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 653945, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 11209 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919426, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:16,805 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 126392 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 722944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,867 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630592_1099518555222] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 9727 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 377307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:11:16,867 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:16,867 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:16,867 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:16,867 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:16,867 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:16,868 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 126410 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967832, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,869 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 11250 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1059818, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:16,973 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 126417 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 855408, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:16,975 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 11257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1056024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:17,089 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 126462 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 820598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:17,173 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 11347 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 968596, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:17,181 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 126494 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 776477, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:17,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 126507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801727, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:17,261 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 11374 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 990259, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:17,321 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 126539 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 832308, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:17,339 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 11396 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972047, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:17,350 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 126546 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 788262, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:17,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 11437 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:17,582 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630592_1099518555222] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 13725 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 458066, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:11:17,582 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:17,582 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:17,582 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:17,584 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 126596 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 876727, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:17,584 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 11451 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946263, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:17,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 11459 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 876575, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:17,788 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 126703 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 944945, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:18,137 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 11793 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1034965, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:18,137 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630592_1099518555222] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 16748 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 520222, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:11:18,137 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 208ms (threshold=20ms); ack: seqno: 126870 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1149685, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:18,138 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:18,138 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:18,138 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:18,138 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:18,138 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:18,187 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630590_1099518555220] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 126878 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812163, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:11:18,467 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 127183 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909497, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:18,529 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 127188 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 940809, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:18,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 12247 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 970522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:18,592 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 127223 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672877, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:18,592 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630592_1099518555222] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 18900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 578533, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:11:18,592 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:18,592 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:18,592 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:18,592 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:18,592 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:18,660 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 127230 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1045750, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:18,668 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 12260 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1151265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:18,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630592_1099518555222] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 18989 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 656000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:11:18,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 12356 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1138018, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:18,823 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 127332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790149, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:18,854 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 12378 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1007809, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:18,931 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 12398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174268, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:19,010 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 12420 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 829140, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:19,039 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 127474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905745, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:19,091 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 12451 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910658, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:19,161 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630591_1099518555221] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 12472 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 816183, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:11:19,262 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:19,262 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:19,262 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:19,263 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630592_1099518555222] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 21674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 664049, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:11:19,295 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 127666 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847451, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:19,492 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 127785 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741664, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:19,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 127875 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 821224, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:19,939 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 128005 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 886991, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:19,939 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:11:19,939 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630592_1099518555222] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 25536 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 473418, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:11:19,939 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:11:19,939 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:11:19,939 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:11:19,939 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:11:20,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 128013 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 955851, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:20,159 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 128046 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:20,184 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 128052 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835802, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:20,219 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 128060 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 848307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:20,294 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 12590 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:20,342 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 128098 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 799863, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:20,436 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 128151 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 752529, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:20,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 12698 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1114254, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:20,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 128205 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 678279, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:20,523 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 12709 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037111, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:20,570 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 12713 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:20,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 128260 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 721623, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:20,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630592_1099518555222] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 29326 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 524226, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:11:20,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 128282 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 735084, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:20,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 12751 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790553, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:20,639 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:20,639 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:20,639 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:20,639 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:20,639 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:20,695 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 128313 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 814748, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:20,704 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 12756 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934234, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:20,775 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 128365 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 668233, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:20,855 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 128418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 803253, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:20,907 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 12873 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 970892, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:20,937 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 128470 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 712690, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:21,018 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 128518 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787929, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:21,099 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 128574 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 706493, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:21,159 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 128601 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 667073, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:21,159 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630592_1099518555222] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 32101 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 399849, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:11:21,159 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:21,159 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:21,159 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:21,159 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:21,159 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:21,190 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 13127 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1036509, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:21,269 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 128680 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 727122, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:21,467 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 13377 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1074303, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:21,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630592_1099518555222] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 34441 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 498477, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:11:21,608 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:21,608 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:21,608 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:21,608 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:21,608 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:21,609 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 129018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1032173, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:21,612 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 13496 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3587185, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:21,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 13504 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026101, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:21,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 145ms (threshold=20ms); ack: seqno: 129065 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919344, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:21,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 13626 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1003964, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:21,920 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630593_1099518555223] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 129097 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908585, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:22,108 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 129215 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062331, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:22,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 198ms (threshold=20ms); ack: seqno: 13751 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1152319, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:22,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 13828 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841242, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:22,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630592_1099518555222] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 37801 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 596656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:11:22,215 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:22,215 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:22,215 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:22,215 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:22,216 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:22,217 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 129230 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946924, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:22,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 13834 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1173528, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:22,289 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 129238 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936497, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:22,469 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:11:22,492 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 205ms (threshold=20ms); ack: seqno: 13843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1294549, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:22,499 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026274961 with entries=77802, filesize=121.97 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026282469 2015-10-16 13:11:22,655 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 13966 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 877924, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:22,764 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 272ms (threshold=20ms); ack: seqno: 129518 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047166, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:22,811 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 1612 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 600850, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:22,811 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 129531 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:22,812 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:22,812 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:22,812 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:22,812 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:22,812 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:22,813 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 14102 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1211199, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:22,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 129537 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 960934, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:22,981 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 160ms (threshold=20ms); ack: seqno: 14111 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948687, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:23,054 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 2298 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865441, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:23,082 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 129673 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945362, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:23,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 129685 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 929347, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:23,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 14334 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 959003, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:23,379 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 129832 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 823521, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:23,537 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 4561 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 795625, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:23,538 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 129953 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 935825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:23,538 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 261ms (threshold=20ms); ack: seqno: 14340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207318, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:23,657 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 14355 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 823522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:23,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 14474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 856615, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:23,825 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 165ms (threshold=20ms); ack: seqno: 130094 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 951928, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:23,863 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630594_1099518555224] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 14475 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 787812, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:23,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 130148 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908863, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 130155 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862718, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,115 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 130245 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 700321, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,115 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 7542 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 603261, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:24,115 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:11:24,116 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 33ms (threshold=20ms) 2015-10-16 13:11:24,116 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:11:24,115 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:11:24,115 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:11:24,177 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 130253 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905472, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,283 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 235ms (threshold=20ms); ack: seqno: 14656 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1067473, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 130397 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 733553, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,406 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 14688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 773711, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,468 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 14732 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 810923, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,527 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 14736 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895636, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,561 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 153ms (threshold=20ms); ack: seqno: 130523 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 817400, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 130534 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 761448, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,648 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 14797 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 876424, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,710 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 14801 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864119, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 130537 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,799 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 11206 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 437595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:24,799 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:11:24,799 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:24,799 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:24,799 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:11:24,799 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:11:24,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 14818 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825988, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 130546 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 823843, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,871 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 130557 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975073, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 14841 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 130564 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857058, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,917 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 14846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 786138, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 130571 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 803878, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:24,959 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 130575 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 872894, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,015 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 130580 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 882937, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 14883 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811121, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,111 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 14888 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 803635, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,169 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 14904 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 803580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,178 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 130667 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 892337, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,277 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 14928 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902942, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,325 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 130802 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758103, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,382 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 14971 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866473, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,390 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 130812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 799320, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,424 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 130820 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866543, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,473 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 14829 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 523443, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:25,473 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:25,473 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:25,473 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:25,473 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:11:25,473 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:11:25,474 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 15019 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,504 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 15027 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 803029, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,560 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 130831 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 829152, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,606 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 130838 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 757184, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,687 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 15168 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 809614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,746 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 15181 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835251, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,811 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 15204 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857777, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,835 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 157ms (threshold=20ms); ack: seqno: 130934 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844813, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,888 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 15237 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 817002, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,895 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 130943 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862085, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,944 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 15269 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 858810, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:25,969 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 130984 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835375, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,030 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 15280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 800449, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 130992 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 799557, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,156 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 18281 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 578693, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:26,156 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:26,156 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:26,156 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:26,156 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:26,156 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:26,158 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 131012 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 880930, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,158 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 15362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 879135, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,199 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 131027 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844363, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,254 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 15391 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887964, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,263 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 131034 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932854, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,354 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 131039 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 970872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,359 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 15434 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782250, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,409 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 15446 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 802771, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,480 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 15464 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 749662, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 131090 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887113, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,553 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 131094 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 893910, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,606 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 15520 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 786837, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,615 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 131102 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1002036, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,670 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 131109 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 823444, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,698 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 15582 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 987455, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,788 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 15676 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 713401, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 21900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 547169, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:26,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 15706 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 678177, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,850 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:26,850 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:26,850 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:26,850 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:26,850 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:11:26,888 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 197ms (threshold=20ms); ack: seqno: 131140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862896, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,962 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 15804 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 759343, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:26,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 131145 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830192, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:27,102 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 15921 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 803730, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:27,140 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 131190 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 805594, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:27,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 16010 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:27,341 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 23871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 679519, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:27,351 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 16094 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 749953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:27,448 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 16167 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 824269, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:27,558 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 24817 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 502347, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:27,558 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 16244 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 735288, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:27,558 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:11:27,558 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:11:27,558 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:11:27,558 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:11:27,558 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:11:28,191 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 1050ms (threshold=20ms); ack: seqno: 131191 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050356332, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:28,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 30382 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 568289, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:28,587 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:11:28,587 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:11:28,587 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:11:28,587 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:11:28,587 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:11:28,663 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 453ms (threshold=20ms); ack: seqno: 131252 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1176601, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:28,664 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 1096ms (threshold=20ms); ack: seqno: 16257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1130487, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:29,061 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 145ms (threshold=20ms); ack: seqno: 32261 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1068983, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:29,061 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 145ms (threshold=20ms) 2015-10-16 13:11:29,061 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 146ms (threshold=20ms) 2015-10-16 13:11:29,061 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 146ms (threshold=20ms) 2015-10-16 13:11:29,061 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 146ms (threshold=20ms) 2015-10-16 13:11:29,061 INFO [sync.2] wal.FSHLog: Slow sync cost: 145 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:29,061 INFO [sync.4] wal.FSHLog: Slow sync cost: 145 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:29,061 INFO [sync.0] wal.FSHLog: Slow sync cost: 145 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:29,061 INFO [sync.3] wal.FSHLog: Slow sync cost: 145 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:29,061 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 146ms (threshold=20ms) 2015-10-16 13:11:29,061 INFO [sync.1] wal.FSHLog: Slow sync cost: 145 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:29,570 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 891ms (threshold=20ms); ack: seqno: 131259 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1716046, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:29,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 862ms (threshold=20ms); ack: seqno: 16333 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1030292, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:29,646 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 35588 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 491645, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:29,646 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:11:29,646 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:11:29,646 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:11:29,646 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:11:29,646 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:11:29,650 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 16337 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1368766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:29,988 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 270ms (threshold=20ms); ack: seqno: 16427 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1116655, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:30,057 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 16433 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1136839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:30,548 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 975ms (threshold=20ms); ack: seqno: 131267 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2377525, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:30,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 788ms (threshold=20ms); ack: seqno: 16513 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1332869, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:30,984 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 433ms (threshold=20ms); ack: seqno: 131274 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2421729, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:31,570 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 647ms (threshold=20ms); ack: seqno: 16529 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1907193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:32,012 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630595_1099518555225] hdfs.DFSClient: Slow ReadProcessor read fields took 1024ms (threshold=20ms); ack: seqno: 131282 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2318652, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:32,032 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 448ms (threshold=20ms); ack: seqno: 16549 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1176903, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:32,163 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630598_1099518555228] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 131299 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1461688, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:32,167 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 16572 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1115818, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:32,505 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630598_1099518555228] hdfs.DFSClient: Slow ReadProcessor read fields took 334ms (threshold=20ms); ack: seqno: 131310 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1799156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:32,510 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 328ms (threshold=20ms); ack: seqno: 16594 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088515, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:32,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630598_1099518555228] hdfs.DFSClient: Slow ReadProcessor read fields took 335ms (threshold=20ms); ack: seqno: 131323 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1949522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:32,853 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630597_1099518555227] hdfs.DFSClient: Slow ReadProcessor read fields took 327ms (threshold=20ms); ack: seqno: 16616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1033898, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:33,013 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630598_1099518555228] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 131446 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 884257, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:33,015 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 16709 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1277457, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:33,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 16752 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 772557, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:33,221 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 16811 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 959736, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:33,296 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 16864 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1084054, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:33,677 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 372ms (threshold=20ms); ack: seqno: 16880 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1227917, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:33,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630598_1099518555228] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 132629 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:34,055 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 374ms (threshold=20ms); ack: seqno: 16887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1294358, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:34,149 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630598_1099518555228] hdfs.DFSClient: Slow ReadProcessor read fields took 220ms (threshold=20ms); ack: seqno: 132741 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1357201, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:34,209 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 16986 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043625, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:34,295 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 17002 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048511, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:34,451 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 17051 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1137322, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:34,560 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 17072 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1045569, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:34,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 17094 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984045, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:34,775 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 17116 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932943, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:34,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 17139 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019030, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:34,919 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 17140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701635, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:34,921 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 133728 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921155, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,020 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 17161 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 893677, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,075 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 17168 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 990851, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 133869 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1051010, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,142 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 17195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 851341, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 17201 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913609, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,197 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 133951 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1311909, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 17224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 118ms (threshold=20ms); ack: seqno: 134004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1080723, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,421 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 17361 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1051593, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,424 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 134056 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908957, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,498 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 17385 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974374, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,574 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 17410 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911368, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,586 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 134210 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 806187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 17433 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917863, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 17436 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1129129, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,763 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 134266 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,763 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 17489 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896305, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,792 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 17493 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857026, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:35,925 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 17583 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1100676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,016 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 193ms (threshold=20ms); ack: seqno: 134342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1007156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,083 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 17690 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963385, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 17697 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 996965, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,144 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 134422 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,229 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 134478 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 646325, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,231 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 17746 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979694, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,302 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 17771 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 901856, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,359 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 17776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 150ms (threshold=20ms); ack: seqno: 134571 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1068000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,497 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 134581 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020330, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,529 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 17882 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1029752, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,578 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 17887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863986, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,636 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 134623 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 878225, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,663 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 17957 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1229104, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,713 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 134676 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 925204, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,727 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 17965 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1113300, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,763 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 17979 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914773, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,823 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 17984 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947534, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,871 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 134775 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 897227, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,905 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 18046 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 939238, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 18050 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979714, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:36,971 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 134782 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964510, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,024 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 134821 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 881258, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,037 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 18075 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 973770, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,104 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 18081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 968860, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 137ms (threshold=20ms); ack: seqno: 134826 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975687, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,203 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 18141 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 973215, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,227 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 134854 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885430, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 134860 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,336 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 134868 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,419 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 18303 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,521 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 154ms (threshold=20ms); ack: seqno: 134903 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758760, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 134908 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864467, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,642 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 18548 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 828333, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,712 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 18552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 12809363, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,766 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 18597 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 659784, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,767 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 134ms (threshold=20ms); ack: seqno: 134987 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020793, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,822 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 18606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 891828, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,826 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 134994 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914970, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,877 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 18627 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 848261, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:37,935 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630599_1099518555229] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 18632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1030493, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:38,045 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 175ms (threshold=20ms); ack: seqno: 135059 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950022, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:38,073 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 135066 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 849728, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:38,186 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 135137 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 951091, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:38,222 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 135141 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 816429, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:38,293 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 135191 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 810092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:38,350 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 135195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865181, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:38,509 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 135244 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983812, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:38,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 18841 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044092, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:38,578 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 135268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 784879, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:38,760 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630600_1099518555230] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 135424 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835910, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:11:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.2857142857142858, tune compaction throughput to unlimited 2015-10-16 13:11:38,968 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 19298 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 960502, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:38,969 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 153ms (threshold=20ms); ack: seqno: 135509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060205, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:39,044 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 135529 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863116, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:39,093 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 19398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790695, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:39,093 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 135544 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 602804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:39,155 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 19406 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 849979, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:39,184 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 135625 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 738750, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:39,256 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 19479 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038989, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:39,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 135626 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919982, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:39,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 19525 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1169605, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:39,416 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 135712 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 876757, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:39,428 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 19688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 713142, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:39,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 135730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912126, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:39,561 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 19853 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946902, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:39,662 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 135783 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1000372, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:39,763 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 20009 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950033, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:39,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 135935 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019717, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:40,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 20169 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989010, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:40,049 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 20180 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 733810, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:40,051 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 136113 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 897060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:40,097 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 136126 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:40,198 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 20247 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1028385, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:40,299 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 136286 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947912, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:40,353 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 20393 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 876076, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:40,464 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 136359 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907469, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:40,542 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 20547 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 860585, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:40,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 136472 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 786050, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:40,754 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 20711 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006370, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:40,865 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 20728 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1042326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:40,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630601_1099518555231] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 20828 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 684979, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:40,969 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 136813 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1033983, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:41,061 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 136824 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 849151, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:41,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 20976 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044777, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:41,135 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 136895 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1009802, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:41,239 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 20994 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1033030, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:41,338 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 137060 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845263, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:41,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 137083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898873, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:41,545 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 137100 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 637479, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:41,547 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 21303 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:41,621 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 137111 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 938471, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:41,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 177ms (threshold=20ms); ack: seqno: 21310 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1253468, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:41,832 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 21375 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1121919, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:41,940 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 21397 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 918508, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:41,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 217ms (threshold=20ms); ack: seqno: 137268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1064513, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 137274 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 961710, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,011 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 21405 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1024832, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,059 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 137279 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888646, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,110 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 137286 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841142, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 21522 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1053389, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,214 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 21528 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 935674, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,234 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 137300 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887130, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,276 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 21545 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847985, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,294 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 137305 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895900, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,312 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 21550 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,376 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 21568 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101737, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,383 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 137319 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 789025, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,409 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 21574 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963745, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,472 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 21592 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 931947, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,487 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 137338 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870687, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,504 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 21596 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1033170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,525 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 137347 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 761997, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 21617 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 975566, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,590 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 137354 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 851143, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,614 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 21624 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1184464, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,626 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 137361 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 832364, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,660 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 21633 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 794425, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,705 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 137372 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 705192, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,719 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 21640 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920302, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,779 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 21655 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 958738, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 137379 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922746, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,824 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 21658 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1025948, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,824 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 137390 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 759715, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,868 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 137395 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864278, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,869 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 21677 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830861, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,934 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 137401 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 890037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:42,944 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 21684 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008153, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:42,997 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 137407 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005057, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:43,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 21701 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 970038, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:43,050 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 137414 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038036, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:43,050 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 21706 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1003794, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:43,089 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 21722 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1024458, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:43,108 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 137418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926396, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:43,170 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 21728 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1229943, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:43,237 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 137425 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1003127, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:43,260 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 21812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023441, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:43,323 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 137437 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1346786, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:43,332 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 21817 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1200498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:43,398 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 137442 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 860950, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:43,433 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 137449 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948798, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:43,523 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 137454 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954080, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:43,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 21992 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2005636, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:43,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 137467 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1335483, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:43,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 21997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995844, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:43,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 137474 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1095290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:43,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 22081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827584, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:43,730 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 137484 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1000664, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:43,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 22087 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:43,809 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 137490 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 856796, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:43,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 22186 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 957968, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:44,091 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 115ms (threshold=20ms); ack: seqno: 22342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1120953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:44,275 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 22504 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1030597, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:44,463 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 22662 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1067519, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:44,568 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 22746 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729912, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:44,629 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 22751 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1083805, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:44,771 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630603_1099518555233] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 22876 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792434, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:11:45,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630602_1099518555232] hdfs.DFSClient: Slow ReadProcessor read fields took 1237ms (threshold=20ms); ack: seqno: 137542 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1236216967, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:45,080 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#53 19227545/10161860 (189.21%), rate=36829.33 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=unlimited, activeCompactions=2] 2015-10-16 13:11:45,260 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630604_1099518555234] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 23097 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907059, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:11:45,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630604_1099518555234] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 24097 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947559, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:11:45,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630605_1099518555235] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 137552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4045021, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:11:46,089 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630604_1099518555234] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 24179 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 891801, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:11:46,191 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630605_1099518555235] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 137843 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862152, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:11:46,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630604_1099518555234] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 24361 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 744609, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:11:46,423 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630604_1099518555234] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 24452 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 805087, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:11:46,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630605_1099518555235] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 138163 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 738367, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:11:46,679 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630604_1099518555234] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 24687 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833505, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:11:46,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630605_1099518555235] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 138345 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 700427, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:11:46,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630604_1099518555234] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 24725 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 680467, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:11:47,000 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630604_1099518555234] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 24895 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1001213, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:11:47,101 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630605_1099518555235] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 138618 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 858612, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:11:47,191 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630606_1099518555236] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 25063 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1004505, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:11:47,241 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#58 average throughput is 29.32 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 13:11:47,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630605_1099518555235] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 138761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779139, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:11:47,264 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/c789fe3f79ef4ea299e01e57a1642c70 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c789fe3f79ef4ea299e01e57a1642c70 2015-10-16 13:11:47,286 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 17091ms (threshold=20ms); ack: seqno: 38760 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 8213768, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:47,286 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:11:47,292 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:11:47,297 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c10e9a74381840e98d2e2bb782a18e3c, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/c10e9a74381840e98d2e2bb782a18e3c 2015-10-16 13:11:47,305 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5189727ab45448b8ae80650ecbc4b00d, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/5189727ab45448b8ae80650ecbc4b00d 2015-10-16 13:11:47,314 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1cab8de30c14490eadfade842e35c3b7, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1cab8de30c14490eadfade842e35c3b7 2015-10-16 13:11:47,322 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/44378bf72e9147e4b8e8577fd8045ba3, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/44378bf72e9147e4b8e8577fd8045ba3 2015-10-16 13:11:47,330 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/63aea3d2bfbc43aabed3ac0887732da7, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/63aea3d2bfbc43aabed3ac0887732da7 2015-10-16 13:11:47,330 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 5 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into c789fe3f79ef4ea299e01e57a1642c70(size=1.5 G), total size for store is 17.6 G. This selection was in queue for 0sec, and took 51sec to execute. 2015-10-16 13:11:47,330 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=5, fileSize=1.5 G, priority=-2, time=36904164761064049; duration=51sec 2015-10-16 13:11:47,330 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:11:47,331 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 8 store files, 7 compacting, 1 eligible, 10 blocking 2015-10-16 13:11:47,331 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:11:47,331 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:11:47,331 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:11:47,597 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630605_1099518555235] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 139178 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:11:47,711 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630605_1099518555235] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 139295 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 688799, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:11:47,849 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630605_1099518555235] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 139457 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 737907, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:11:47,931 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536925696 > lower bound=16777216 2015-10-16 13:11:47,931 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 13:11:48,037 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 750ms (threshold=20ms); ack: seqno: 38761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862615, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:48,042 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 139628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1527643, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:48,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 165ms (threshold=20ms); ack: seqno: 139632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1588121, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:48,286 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 139639 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1325488, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:48,334 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 139644 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1237287, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:48,416 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 139650 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1384387, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:48,538 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 139662 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1221747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:48,663 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 139674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060569, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:48,811 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 139692 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992760, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:48,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 139697 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 987763, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:48,997 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 139782 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019575, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:49,057 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 139787 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1041458, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:49,108 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 139795 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948774, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:49,167 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 139799 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 879568, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:49,242 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630608_1099518555238] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 1788 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 698631, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:49,262 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 139810 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 925801, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:49,367 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 139825 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 896116, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:49,425 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 139831 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 815982, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:49,496 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 139837 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866428, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:49,556 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 139842 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010958, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:49,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 139850 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1035946, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:49,687 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 139860 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2704825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:49,758 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 139865 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1120650, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:49,982 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 139997 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1064858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:50,161 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 140122 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:50,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 140259 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1034227, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:50,413 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 140298 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 918246, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:50,414 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630609_1099518555239] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 3673 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:11:51,038 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=12049703, memsize=512.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/7d42478229094befa163faabeb602859 2015-10-16 13:11:51,044 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/7d42478229094befa163faabeb602859 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7d42478229094befa163faabeb602859 2015-10-16 13:11:51,050 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7d42478229094befa163faabeb602859, entries=1864380, sequenceid=12049703, filesize=283.2 M 2015-10-16 13:11:51,051 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 3014ms (threshold=20ms); ack: seqno: 38762 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 518375, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:51,051 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536925280, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 3120ms, sequenceid=12049703, compaction requested=false 2015-10-16 13:11:51,104 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630607_1099518555237] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 141219 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 677403, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:11:51,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630611_1099518555241] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 142312 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 707187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:52,100 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630611_1099518555241] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 142583 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 714032, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:52,870 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630611_1099518555241] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 143719 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 668621, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:11:53,055 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 143955 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 646792, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:53,326 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 164ms (threshold=20ms); ack: seqno: 144088 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062778, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:53,485 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 144247 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966943, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:53,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 144409 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874699, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:53,849 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 202ms (threshold=20ms); ack: seqno: 144416 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1045523, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:53,917 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 144422 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1163460, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,092 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 144574 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941817, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,153 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 144581 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911422, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,186 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 144592 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920733, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 144599 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,293 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 144610 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 988324, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,355 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 144619 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006828, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,396 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 144629 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1148718, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,458 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 144633 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1119127, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,499 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 144641 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011320, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,558 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 144645 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043029, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,603 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 144652 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863301, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,666 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 144660 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,718 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 144672 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948845, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 144676 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1488643, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,834 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 144683 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 998447, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,880 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 144688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1007786, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:54,951 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 144730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813978, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:55,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 144784 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 764572, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:55,144 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 144833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 851849, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:55,240 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 144885 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:55,335 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 144938 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:55,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 144988 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 793444, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:55,527 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 145042 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861774, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:55,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 145095 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 831802, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:55,736 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 145146 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782260, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:55,859 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 145195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012992, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:56,013 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 145273 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1040121, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:56,084 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630612_1099518555242] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 145321 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 789468, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:11:57,042 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630613_1099518555243] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 146696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729642, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:11:57,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630613_1099518555243] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 147526 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 918138, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:11:58,016 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630614_1099518555244] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 148068 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 723497, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:58,351 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630596_1099518555226] hdfs.DFSClient: Slow ReadProcessor read fields took 7300ms (threshold=20ms); ack: seqno: 38763 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741419, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:58,409 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630614_1099518555244] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 148636 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790649, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:58,506 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:11:58,538 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026282469 with entries=77838, filesize=122.02 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026318506 2015-10-16 13:11:58,540 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026171960 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026171960 2015-10-16 13:11:58,546 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026274961 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026274961 2015-10-16 13:11:58,720 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630614_1099518555244] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 149004 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 689127, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:58,720 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 847 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 517847, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:11:58,720 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:58,720 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:58,720 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:58,720 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:58,720 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:11:59,231 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630614_1099518555244] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 149650 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 647900, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:11:59,231 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:59,231 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 3698 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 453233, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:11:59,231 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:59,231 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:59,231 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:59,231 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:11:59,538 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 4823 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1115229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:11:59,736 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630616_1099518555246] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 150301 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 858833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:11:59,829 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 6211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 436386, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:11:59,829 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630616_1099518555246] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 150369 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2886207, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:11:59,829 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:11:59,829 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:11:59,829 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:11:59,829 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:11:59,829 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:12:00,368 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 9116 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 492404, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:12:00,368 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630616_1099518555246] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 151026 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 670861, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:00,368 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:12:00,368 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:12:00,368 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:12:00,368 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:12:00,368 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:12:00,903 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 11960 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 440382, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:12:00,903 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630616_1099518555246] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 151688 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 679187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:00,903 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:12:00,903 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:12:00,903 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:12:00,903 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:12:00,903 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:12:01,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 14732 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 415431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:12:01,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630617_1099518555247] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 152354 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 807062, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:01,443 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:12:01,443 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:12:01,443 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:12:01,443 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:12:01,443 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 41ms (threshold=20ms) 2015-10-16 13:12:01,535 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 15089 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 450099, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:12:01,625 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630617_1099518555247] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 152541 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1051539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:02,022 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630617_1099518555247] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 153001 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 756686, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:02,022 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 17586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 415767, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:12:02,022 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:02,022 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:02,022 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:02,022 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:02,022 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:02,564 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630617_1099518555247] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 153646 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 757732, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:02,564 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 20531 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 449834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:12:02,564 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:02,564 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:02,564 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:02,564 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:02,564 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:02,951 DEBUG [PriorityRpcServer.handler=1,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:12:02,951 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134267376 > lower bound=16777216 2015-10-16 13:12:02,966 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.41 MB 2015-10-16 13:12:02,991 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 22648 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 690423, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:12:03,124 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630619_1099518555249] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 154297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 849371, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:03,124 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630618_1099518555248] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904117, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:03,124 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 23173 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 594719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:12:03,124 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:12:03,124 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:12:03,124 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:12:03,124 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:12:03,124 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:12:03,482 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 127ms (threshold=20ms); ack: seqno: 24392 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 609943, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:12:03,603 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630618_1099518555248] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 836 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 817106, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:03,603 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 24841 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 636776, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:12:03,603 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630619_1099518555249] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 154907 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866264, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:03,603 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:12:03,603 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:12:03,603 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:12:03,603 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:12:03,603 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:12:03,829 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=12096483, memsize=128.5 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ea56a0f385924c7b9c4508e67d94fbcd 2015-10-16 13:12:03,834 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/ea56a0f385924c7b9c4508e67d94fbcd as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ea56a0f385924c7b9c4508e67d94fbcd 2015-10-16 13:12:03,840 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ea56a0f385924c7b9c4508e67d94fbcd, entries=467760, sequenceid=12096483, filesize=71.1 M 2015-10-16 13:12:03,842 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.47 MB/134710480, currentsize=19.00 MB/19920480 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 875ms, sequenceid=12096483, compaction requested=true 2015-10-16 13:12:03,842 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:12:03,842 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 7 compacting, 3 eligible, 10 blocking 2015-10-16 13:12:03,843 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 13:12:03,843 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:12:03,843 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:12:04,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 27403 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 756727, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:12:04,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630619_1099518555249] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 155456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 740906, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:04,129 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:12:04,129 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:04,129 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:12:04,129 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:12:04,129 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:12:04,680 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630619_1099518555249] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 156118 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 671094, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:04,680 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 30225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 535378, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:12:04,681 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:12:04,681 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:12:04,681 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:12:04,681 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:12:04,681 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 39ms (threshold=20ms) 2015-10-16 13:12:04,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 156448 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895556, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:05,131 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 172ms (threshold=20ms); ack: seqno: 156453 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 990450, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:05,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 193ms (threshold=20ms); ack: seqno: 156502 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022466, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:05,411 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 34139 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 816813, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:12:05,411 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:05,411 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 156515 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847846, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:05,411 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:05,411 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:05,411 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:05,411 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:05,469 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 156526 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1023371, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:05,513 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 156552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 848232, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:05,580 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 156561 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 865912, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:05,669 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 156577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:05,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 156625 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963043, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:05,805 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 156632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1091280, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:05,887 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 156678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020418, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:05,919 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 156687 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022673, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:06,050 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 156741 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1391114, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:06,122 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 156786 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971676, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:06,225 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630615_1099518555245] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 38456 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 485000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:12:06,225 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 156798 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 959702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:06,225 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:12:06,225 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:12:06,225 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:12:06,225 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:12:06,262 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 156809 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963801, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:06,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:12:06,304 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026318506 with entries=77830, filesize=122.01 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026326273 2015-10-16 13:12:06,306 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026282469 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026282469 2015-10-16 13:12:06,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 156834 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037283, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:06,428 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 156869 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1081705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:06,470 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 156875 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956529, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:06,541 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 156919 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950542, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:06,598 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 156931 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1067013, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:06,672 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 156972 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1122253, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:06,724 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 156979 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012866, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:06,806 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 157022 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 820954, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:06,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 157032 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017714, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:06,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 157074 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1007599, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:07,021 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630621_1099518555251] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 3862 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 612045, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:07,021 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 157119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 698275, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:07,021 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:12:07,021 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:12:07,021 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:12:07,021 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:12:07,021 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 42ms (threshold=20ms) 2015-10-16 13:12:07,234 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630621_1099518555251] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 4318 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969091, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:07,645 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630620_1099518555250] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 157878 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 660655, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:12:07,645 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630621_1099518555251] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 6311 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 553159, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:07,645 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:12:07,645 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:12:07,645 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:12:07,645 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:12:07,645 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:12:07,796 DEBUG [PriorityRpcServer.handler=0,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:12:07,796 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134264736 > lower bound=16777216 2015-10-16 13:12:07,796 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=128.02 MB 2015-10-16 13:12:07,820 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630621_1099518555251] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 7107 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 711950, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:08,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630621_1099518555251] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 8616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 621766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:08,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 158478 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1594852, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:08,152 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:08,152 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:12:08,152 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 38ms (threshold=20ms) 2015-10-16 13:12:08,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630622_1099518555252] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 375 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 719063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:08,152 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:08,599 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630622_1099518555252] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 1016 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 740379, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:08,599 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630621_1099518555251] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 10676 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 523607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:08,600 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:08,600 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:08,600 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:08,600 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:08,600 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:08,695 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=12143117, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1b561b85de884031b2d1488825032232 2015-10-16 13:12:08,712 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/1b561b85de884031b2d1488825032232 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1b561b85de884031b2d1488825032232 2015-10-16 13:12:08,724 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1b561b85de884031b2d1488825032232, entries=466300, sequenceid=12143117, filesize=70.8 M 2015-10-16 13:12:08,725 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134290240, currentsize=23.48 MB/24623120 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 929ms, sequenceid=12143117, compaction requested=true 2015-10-16 13:12:08,725 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:12:08,725 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 7 compacting, 4 eligible, 10 blocking 2015-10-16 13:12:08,727 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio 2015-10-16 13:12:08,727 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:12:08,727 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:12:09,104 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 577ms (threshold=20ms); ack: seqno: 158986 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 577536025, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:09,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630621_1099518555251] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 14699 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 403849, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:09,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 159324 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 720032, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:09,364 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:09,364 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 36ms (threshold=20ms) 2015-10-16 13:12:09,364 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:09,364 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:09,364 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 37ms (threshold=20ms) 2015-10-16 13:12:09,786 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 151ms (threshold=20ms); ack: seqno: 159683 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 759924, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:09,923 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 159689 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 900774, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:09,999 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 159701 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779898, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:10,084 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 159748 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 688902, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:10,084 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630621_1099518555251] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 18509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 598775, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:10,084 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:12:10,084 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:12:10,084 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:12:10,084 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:12:10,084 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:12:10,146 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 159756 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 854784, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:10,227 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 159827 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 897026, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:10,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 159833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 858147, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:10,378 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 159853 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1203709, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:10,441 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 159864 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 849326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:10,505 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 159872 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 6322475, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:10,588 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 159888 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012589, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:10,632 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 159898 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950686, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:10,688 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 159904 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017646, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:10,737 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 159911 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1032908, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:10,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 159916 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 966522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:10,839 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 159923 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 872292, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:10,892 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 159928 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947846, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:10,959 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 159936 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992451, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:11,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630621_1099518555251] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 23265 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 489391, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:11,007 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:12:11,007 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40ms (threshold=20ms) 2015-10-16 13:12:11,009 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 159943 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910259, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:11,082 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 159950 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888584, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:11,166 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 159961 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1223622, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:11,197 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 159967 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 829188, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:11,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 159974 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930115, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:11,373 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 159986 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 949281, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:11,518 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 160041 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 850045, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:11,758 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630621_1099518555251] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 26877 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 580911, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:11,837 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630621_1099518555251] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 27094 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 485169, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:11,837 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:12:11,837 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:12:11,837 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:12:11,837 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:12:11,837 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 44ms (threshold=20ms) 2015-10-16 13:12:11,910 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 152ms (threshold=20ms); ack: seqno: 160357 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 832157, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:12,049 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630623_1099518555253] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 160412 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 880665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:12,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630624_1099518555254] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 160545 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934495, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:12,431 DEBUG [PriorityRpcServer.handler=4,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:12:12,431 WARN [MemStoreFlusher.0] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 13:12:12,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:12:12,431 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 7 compacting, 4 eligible, 10 blocking 2015-10-16 13:12:12,433 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio 2015-10-16 13:12:12,433 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:12:12,433 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:12:12,445 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630624_1099518555254] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 160681 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1073650, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:12,558 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630621_1099518555251] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 30822 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701746, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:12,558 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630624_1099518555254] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 160783 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 749734, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:12,558 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:12,558 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:12,558 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:12,558 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:12,558 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:12,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630624_1099518555254] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 160791 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 990934, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:12,890 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630624_1099518555254] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 160931 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 883333, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:13,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630624_1099518555254] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 161072 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 824892, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:13,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630624_1099518555254] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 161170 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 998806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:13,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630621_1099518555251] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 34570 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 663308, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:13,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630624_1099518555254] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 161238 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 765275, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:13,284 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:13,284 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:13,284 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:13,284 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:13,284 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 43ms (threshold=20ms) 2015-10-16 13:12:13,348 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630624_1099518555254] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 161247 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 735479, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:13,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630624_1099518555254] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 161380 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 814220, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:13,718 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630624_1099518555254] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 161508 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758096, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:13,805 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630624_1099518555254] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 161592 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 981665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:13,924 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630624_1099518555254] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 161615 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 960222, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:14,001 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630621_1099518555251] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 38500 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 457649, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:12:14,001 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:12:14,001 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:12:14,001 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:12:14,001 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:12:14,001 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:12:14,002 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630624_1099518555254] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 161626 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979030, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:14,078 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:12:14,112 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026326273 with entries=77847, filesize=122.03 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026334078 2015-10-16 13:12:14,115 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026318506 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026318506 2015-10-16 13:12:14,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630625_1099518555255] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 2329 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 483538, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:14,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630624_1099518555254] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 162288 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 696988, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:14,601 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:12:14,601 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:12:14,601 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:12:15,155 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630625_1099518555255] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 5047 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 615652, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:15,155 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 162979 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834171, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:15,155 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:12:15,155 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:12:15,155 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:12:15,155 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 45ms (threshold=20ms) 2015-10-16 13:12:15,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 163616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 746720, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:15,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630625_1099518555255] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 8028 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 467474, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:15,735 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:12:15,735 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:12:15,735 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:12:15,735 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:12:16,110 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 163892 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930552, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:16,180 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 163945 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 761165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:16,256 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 163996 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1287538, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:16,334 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 164044 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 807872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:16,359 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 164051 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811012, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:16,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630625_1099518555255] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 11634 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 380854, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:16,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 164089 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 742665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:16,443 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:12:16,443 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:12:16,443 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:12:16,443 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:12:16,443 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:12:16,541 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 164145 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:16,629 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 164197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859960, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:16,709 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 164248 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 879410, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:16,904 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 187ms (threshold=20ms); ack: seqno: 164259 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1029923, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:17,005 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 164277 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910052, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:17,052 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 164283 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995490, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:17,107 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 164289 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1049227, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:17,189 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630625_1099518555255] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 15360 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 561116, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:17,255 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 164403 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1047078, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:17,325 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630625_1099518555255] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 15896 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 546615, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:17,325 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:12:17,325 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:12:17,326 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:12:17,326 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:12:17,325 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 46ms (threshold=20ms) 2015-10-16 13:12:17,327 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 164407 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1046848, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:17,399 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 164413 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919090, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:17,469 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 164421 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972148, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:17,508 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 164428 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941594, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:17,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 164438 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 978445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:17,724 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630626_1099518555256] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 164528 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1028794, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:17,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630627_1099518555257] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 164671 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1175817, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:18,050 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630625_1099518555255] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 19896 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 535683, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:18,050 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:12:18,050 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:12:18,050 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:12:18,050 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:18,084 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630627_1099518555257] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 164782 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 957311, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:18,271 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630627_1099518555257] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 164923 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 980105, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:18,470 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630627_1099518555257] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 165066 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 882371, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:18,672 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630627_1099518555257] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 165212 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1009741, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:18,751 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630627_1099518555257] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 165256 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 627869, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:18,751 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630625_1099518555255] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 23531 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 502781, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:18,753 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:12:18,837 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630627_1099518555257] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 165263 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1108128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:18,986 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630627_1099518555257] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 165305 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 992222, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:19,211 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630627_1099518555257] hdfs.DFSClient: Slow ReadProcessor read fields took 119ms (threshold=20ms); ack: seqno: 165450 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1052873, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:19,314 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630627_1099518555257] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 165504 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1079897, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:19,472 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:19,472 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:19,472 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630627_1099518555257] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 165648 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 679281, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:19,472 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:19,473 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630625_1099518555255] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 27421 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 657034, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:19,731 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630627_1099518555257] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 165970 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1082238, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:20,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630625_1099518555255] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 30315 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 409450, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:20,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630627_1099518555257] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 166310 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 658562, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:20,031 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:12:20,031 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:12:20,031 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:12:20,695 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630625_1099518555255] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 33904 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 541682, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:20,695 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:12:20,695 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:20,695 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:20,695 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:20,695 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:12:21,487 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 166969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 968486, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:21,631 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630625_1099518555255] hdfs.DFSClient: Slow ReadProcessor read fields took 343ms (threshold=20ms); ack: seqno: 37400 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 925046, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:21,631 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 343ms (threshold=20ms) 2015-10-16 13:12:21,631 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 343ms (threshold=20ms) 2015-10-16 13:12:21,631 INFO [sync.3] wal.FSHLog: Slow sync cost: 343 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:21,632 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 343ms (threshold=20ms) 2015-10-16 13:12:21,632 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 343ms (threshold=20ms) 2015-10-16 13:12:21,632 INFO [sync.1] wal.FSHLog: Slow sync cost: 343 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:21,632 INFO [sync.4] wal.FSHLog: Slow sync cost: 343 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:21,632 INFO [sync.2] wal.FSHLog: Slow sync cost: 343 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:21,632 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 343ms (threshold=20ms) 2015-10-16 13:12:21,632 INFO [sync.0] wal.FSHLog: Slow sync cost: 343 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:21,660 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 167130 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006796, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:21,752 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:12:21,752 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630625_1099518555255] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 37818 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 397031, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK]] 2015-10-16 13:12:21,752 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 167191 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 639952, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:21,753 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:12:21,753 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:12:21,837 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 167195 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920509, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:21,895 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 167207 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982661, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:21,983 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:12:22,028 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026334078 with entries=77941, filesize=122.18 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026341984 2015-10-16 13:12:22,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834360, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,034 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 137ms (threshold=20ms); ack: seqno: 167211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885704, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,090 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 167218 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917163, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,151 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 167223 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1149405, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,195 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 167231 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 916390, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,259 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 167235 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1071205, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,314 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 167243 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942098, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,408 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 167261 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942131, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 167265 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964145, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,520 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 167273 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1159081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,557 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 2346 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838064, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,630 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 167292 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905555, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,691 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 167299 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956547, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,786 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 3455 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 536644, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,786 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:12:22,786 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:12:22,786 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:12:22,786 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:12:22,786 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:12:22,788 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 167310 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1096682, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 167313 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,926 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 167321 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898802, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:22,983 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 167328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:23,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 167340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965413, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:23,143 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 167346 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019930, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:23,198 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 167353 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 996398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:23,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 167358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1015054, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:23,275 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 167365 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1018891, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:23,339 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 167369 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 991991, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:23,614 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 140ms (threshold=20ms); ack: seqno: 167527 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1075404, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:23,614 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 8096 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 413463, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:23,614 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:12:23,614 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:12:23,614 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:12:23,614 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:23,614 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 49ms (threshold=20ms) 2015-10-16 13:12:23,683 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 167534 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936910, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:23,894 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 107ms (threshold=20ms); ack: seqno: 167668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1032063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:24,067 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 167814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013081, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:24,262 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 11908 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 479666, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:24,262 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:24,262 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:24,262 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:24,262 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:24,262 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:24,314 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 143ms (threshold=20ms); ack: seqno: 167943 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1076729, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:24,480 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 168076 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1001508, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:24,650 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 168246 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902162, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:24,736 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 168297 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:24,817 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 168351 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 900748, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:24,940 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 15679 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 425093, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:24,940 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:24,940 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:24,940 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:12:24,940 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 47ms (threshold=20ms) 2015-10-16 13:12:24,940 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:12:24,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 168401 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919291, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:25,085 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 168508 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 990398, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:25,165 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 168562 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862756, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:25,262 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 168614 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 798533, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:25,351 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 168661 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1089332, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:25,444 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 168714 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1046217, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:25,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630628_1099518555258] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 168765 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1056773, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:25,665 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 181ms (threshold=20ms); ack: seqno: 18911 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1070765, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:25,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 19295 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 418223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:25,783 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:12:25,783 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:12:25,783 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:12:25,783 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:12:25,783 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 52ms (threshold=20ms) 2015-10-16 13:12:26,408 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630630_1099518555260] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 169437 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932733, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:26,408 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 22575 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 514859, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:26,408 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:12:26,408 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:12:26,408 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 50ms (threshold=20ms) 2015-10-16 13:12:26,408 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 51ms (threshold=20ms) 2015-10-16 13:12:26,949 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630630_1099518555260] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 170231 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 890337, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:27,342 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630630_1099518555260] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 170713 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 819848, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:28,280 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630631_1099518555261] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 172061 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 631984, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:28,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630631_1099518555261] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 172073 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1098226, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:28,459 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630631_1099518555261] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 172109 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835402, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:28,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630631_1099518555261] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 172268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 872598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:28,783 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630631_1099518555261] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 172429 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943478, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:28,954 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630631_1099518555261] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 172588 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 737020, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:29,126 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630631_1099518555261] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 172739 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885925, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:29,154 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630631_1099518555261] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 172747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902331, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:29,214 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630631_1099518555261] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 172754 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 903258, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:29,398 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630631_1099518555261] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 172907 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932925, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:29,569 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 173073 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 900764, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:29,758 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 173229 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838111, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:29,946 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 173329 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1167572, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:30,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 173333 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1006202, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:30,092 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 173401 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 577569, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:30,161 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 173408 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1059526, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:30,223 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 173455 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 751124, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:30,314 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 173459 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1137941, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:30,445 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 173510 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1155799, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:30,583 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 173568 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 893420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:30,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 173619 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 820546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:30,764 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 173665 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 860200, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:30,846 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 173718 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895073, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:30,936 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 173769 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 846690, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:31,028 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 173822 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866624, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:31,118 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 173871 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812988, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:31,206 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 173923 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974966, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:31,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 173976 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:31,822 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 174750 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 656316, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:31,956 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 174887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 807969, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:32,230 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630632_1099518555262] hdfs.DFSClient: Slow ReadProcessor read fields took 220ms (threshold=20ms); ack: seqno: 174952 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1516516, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:32,384 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 175120 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1066314, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:32,552 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 175280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1176785, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:32,736 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 175436 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1174632, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:32,917 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 175593 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1100920, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:33,103 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 175761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 955322, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:33,296 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 175929 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1130027, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:33,329 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 175936 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982853, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:33,402 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 175946 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038606, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:33,471 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 176018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 997047, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:33,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 176074 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1060483, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:33,672 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 176118 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 650544, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:33,747 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 176125 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1080943, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:33,975 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 176260 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1108192, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:34,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 176387 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019211, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:34,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 176529 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1148347, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:34,486 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 176627 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1135367, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:34,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 176714 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1132395, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:34,857 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 176784 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1147769, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:35,046 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 176906 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1027568, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:35,234 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 177026 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:35,342 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 177117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 801932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:35,422 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630633_1099518555263] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 177121 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:35,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 177216 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1430211, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:35,865 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 177273 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1126788, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:35,955 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 177324 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1099900, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:36,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 177374 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1018851, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:36,102 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 177427 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1082477, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:36,174 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 177466 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 757787, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:36,255 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 177507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 925892, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:36,813 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 515ms (threshold=20ms); ack: seqno: 177561 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1488825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:37,041 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 195ms (threshold=20ms); ack: seqno: 177611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1177506, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:37,378 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 279ms (threshold=20ms); ack: seqno: 177667 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1367607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:37,854 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 178276 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 957161, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:37,907 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 178283 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1210570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:38,150 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 239ms (threshold=20ms); ack: seqno: 178291 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1450403, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:38,529 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 374ms (threshold=20ms); ack: seqno: 178299 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1624714, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 13:12:38,868 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 178679 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 875097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:38,998 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 178816 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 709414, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:39,052 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 178824 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 890367, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:39,214 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 178971 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:39,387 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630634_1099518555264] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 179121 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1128781, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:12:39,568 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 179264 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3283859, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:39,744 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 179400 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1077150, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:39,923 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 179550 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1169797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:40,102 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 179704 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1267764, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:40,205 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 179716 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1184303, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:40,383 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 179870 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1207151, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:40,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 179921 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1114642, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:40,715 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 180182 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 791292, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:40,847 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 180282 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1074505, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:41,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 180453 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1004372, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:41,201 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 180626 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1112528, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:41,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 180788 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941703, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:41,567 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 146ms (threshold=20ms); ack: seqno: 180794 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 980315, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:41,736 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 180943 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 9722082, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:41,894 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 181097 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928524, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:42,072 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630635_1099518555265] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 181257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037414, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:12:42,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630636_1099518555266] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 181421 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1348246, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:42,426 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630636_1099518555266] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 181532 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1204633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:42,481 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630636_1099518555266] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 181548 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 834950, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:42,529 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630636_1099518555266] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 181555 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1084493, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:42,714 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630636_1099518555266] hdfs.DFSClient: Slow ReadProcessor read fields took 83ms (threshold=20ms); ack: seqno: 181647 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:42,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630636_1099518555266] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 181798 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1177487, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:43,083 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630636_1099518555266] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 181954 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928087, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:43,272 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630636_1099518555266] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 182086 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1355304, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:43,495 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630636_1099518555266] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 182194 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 926164, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:43,681 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630636_1099518555266] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 182340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936934, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:43,858 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630636_1099518555266] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 182480 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1420965, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:44,033 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630636_1099518555266] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 182592 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022497, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:44,220 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630636_1099518555266] hdfs.DFSClient: Slow ReadProcessor read fields took 87ms (threshold=20ms); ack: seqno: 182727 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1182886, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:44,407 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630636_1099518555266] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 182875 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1367124, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:44,467 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630636_1099518555266] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 182894 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 755594, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:12:45,079 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#53 38703976/20543496 (188.40%), rate=46943.86 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=unlimited, activeCompactions=1] 2015-10-16 13:12:45,565 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630637_1099518555267] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 184251 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 620930, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:45,969 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630637_1099518555267] hdfs.DFSClient: Slow ReadProcessor read fields took 131ms (threshold=20ms); ack: seqno: 184592 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 973847, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:46,345 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630637_1099518555267] hdfs.DFSClient: Slow ReadProcessor read fields took 209ms (threshold=20ms); ack: seqno: 184747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1217128, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:46,512 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630637_1099518555267] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 184876 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:46,678 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630637_1099518555267] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 185011 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986809, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:46,824 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630637_1099518555267] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 185089 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736117, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:46,892 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630637_1099518555267] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 185096 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1045772, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:47,080 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630637_1099518555267] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 185225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:47,253 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630637_1099518555267] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 185329 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 971163, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:47,432 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630637_1099518555267] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 185462 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1172932, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:12:47,521 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 185542 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 753705, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:47,584 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 185546 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812701, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:47,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 185607 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 596669, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:47,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 185616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 862998, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:47,922 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 185753 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020679, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:48,102 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 185887 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839834, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:48,148 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 185924 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 22062956, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:48,324 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 120ms (threshold=20ms); ack: seqno: 186024 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911872, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:48,504 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 186165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019693, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:48,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 172ms (threshold=20ms); ack: seqno: 186312 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1116002, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:48,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 186462 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 885950, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:49,104 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 186616 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946510, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:49,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 186744 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1164315, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:49,465 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 186842 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054821, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:49,577 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 186929 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 760816, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:49,676 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630638_1099518555268] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 186965 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 774022, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:12:50,776 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630639_1099518555269] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 188342 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 822651, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:50,912 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630639_1099518555269] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 188484 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 939483, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:51,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630639_1099518555269] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 189155 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 788132, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:12:51,860 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 189699 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 613952, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:52,918 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 163ms (threshold=20ms); ack: seqno: 190878 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1220555, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:52,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 190930 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 979993, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:53,052 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 190985 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909639, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:53,121 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 191049 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 783399, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:53,175 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 191072 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 640838, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:53,233 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 191103 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943418, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:53,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 191152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 923044, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:53,392 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 191202 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 768627, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:53,479 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 191255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943891, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:53,566 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 191310 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 949055, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:53,652 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 191359 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 800734, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:53,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 191412 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1025285, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:53,842 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 191467 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1028748, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:53,936 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 191515 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 956737, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:54,029 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 191567 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 948704, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:54,123 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 191620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1000225, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:54,172 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 191656 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 879482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:54,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 191673 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 797222, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:54,305 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 191724 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863751, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:54,607 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630640_1099518555270] hdfs.DFSClient: Slow ReadProcessor read fields took 303ms (threshold=20ms); ack: seqno: 191726 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043298, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:12:55,078 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630641_1099518555271] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 192429 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 711545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:56,022 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630641_1099518555271] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 193792 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 674506, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:12:56,346 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 194162 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 920312, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:56,608 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 158ms (threshold=20ms); ack: seqno: 194327 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 973058, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:56,767 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 194483 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:56,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 219ms (threshold=20ms); ack: seqno: 194492 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1217440, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:57,066 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 194507 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069630, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:57,125 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 194519 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017546, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:57,267 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 194633 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 973075, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:57,329 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 194641 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 958545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:57,511 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 194791 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 941131, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:57,691 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 194937 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 833731, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:57,873 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 195089 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 898654, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:57,937 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 195116 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:57,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 195123 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847810, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:58,059 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 195165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 643123, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:58,127 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 195175 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 840317, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:58,308 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 195330 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 848424, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:58,517 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 195479 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1112237, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:58,767 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 175ms (threshold=20ms); ack: seqno: 195576 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054634, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:58,814 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 195611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 743348, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:58,875 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 195617 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982916, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:59,033 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 195757 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 953061, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:59,885 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630642_1099518555272] hdfs.DFSClient: Slow ReadProcessor read fields took 751ms (threshold=20ms); ack: seqno: 195894 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 749806592, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-c7652f74-fedd-425a-8e97-d4f6fb2babae,DISK]] 2015-10-16 13:12:59,961 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630643_1099518555273] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 195982 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 889307, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:00,163 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630643_1099518555273] hdfs.DFSClient: Slow ReadProcessor read fields took 111ms (threshold=20ms); ack: seqno: 196110 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1124281, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:00,476 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630643_1099518555273] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 196517 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 746989, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:00,620 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630643_1099518555273] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 196702 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969393, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:01,446 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630643_1099518555273] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 197869 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 736650, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:01,948 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630644_1099518555274] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 198597 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 687928, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:13:02,402 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630644_1099518555274] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 199223 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 767313, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:13:02,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630644_1099518555274] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 199485 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 980937, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:13:03,381 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 200595 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 671195, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:03,541 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 200766 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 775824, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:03,874 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 179ms (threshold=20ms); ack: seqno: 201007 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1131512, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:03,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 201083 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983238, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:04,025 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 201133 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1243639, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:04,152 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 201238 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 809481, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:04,237 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 201289 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1080658, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:04,314 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 201339 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 858170, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:04,401 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 201392 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 919496, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:04,488 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 201442 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1017667, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:04,582 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 201496 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1024865, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:04,671 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 201545 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014332, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:04,759 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 201599 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909088, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:04,856 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 201653 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1120782, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:04,947 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 201705 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 880500, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:05,044 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 201761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972837, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:05,141 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 201812 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1073050, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:05,238 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 201866 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1245569, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:05,335 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 201918 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1076859, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:05,389 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 201942 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 785350, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:05,465 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 201971 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013070, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:05,573 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630645_1099518555275] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 202019 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069980, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:06,467 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630646_1099518555276] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 203296 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 658461, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:13:06,559 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630646_1099518555276] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 203398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 760598, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:13:07,432 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630647_1099518555277] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 204663 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 656611, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:13:08,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630647_1099518555277] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 206024 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 676024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:13:08,735 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630648_1099518555278] hdfs.DFSClient: Slow ReadProcessor read fields took 114ms (threshold=20ms); ack: seqno: 206344 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4600973, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:09,508 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630648_1099518555278] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 207397 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729477, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:10,245 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630648_1099518555278] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 208332 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932296, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:10,372 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630649_1099518555279] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 208497 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 903019, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:13:10,592 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630649_1099518555279] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 208754 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 665643, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:13:11,524 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630649_1099518555279] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 210110 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 771827, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK]] 2015-10-16 13:13:12,291 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630650_1099518555280] hdfs.DFSClient: Slow ReadProcessor read fields took 38ms (threshold=20ms); ack: seqno: 211219 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 882317, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:12,520 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630650_1099518555280] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 211485 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:12,849 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630650_1099518555280] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 211956 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:13,472 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 212841 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 726765, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:13,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 213533 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839710, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:14,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 214169 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 42521246, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:14,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 214362 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947501, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:14,799 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 214529 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1195041, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:15,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 205ms (threshold=20ms); ack: seqno: 214541 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1375109, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:15,080 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 214556 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913015, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:15,136 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 214560 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 943696, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:15,182 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 214567 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888681, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:15,243 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 214572 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 906801, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:15,327 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 214583 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1052068, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:15,416 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 214595 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1118459, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:15,509 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 214608 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1001852, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:15,569 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 214614 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974780, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:15,601 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 214620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 950247, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:15,663 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 214631 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924983, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:15,698 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 214635 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 974851, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:15,770 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 214642 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993832, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:15,830 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630651_1099518555281] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 214646 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012167, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:13:15,932 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 214665 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1332727, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:15,993 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 214676 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1315886, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:16,044 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 214683 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1233169, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:16,095 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 214695 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836515, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:16,148 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 214699 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972442, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:16,202 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 214708 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1005692, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:16,258 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 214713 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1187539, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:16,320 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 214725 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1112767, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:16,469 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 214790 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 980903, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:16,647 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 214951 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 903400, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:16,826 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 215114 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1142892, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:16,993 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 102ms (threshold=20ms); ack: seqno: 215154 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1104056, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:17,157 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 157ms (threshold=20ms); ack: seqno: 215162 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1105479, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:17,320 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 215306 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1040545, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:17,482 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 215451 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010382, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:17,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 215560 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 697384, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:17,646 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 215572 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1028716, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:17,819 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 215733 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842467, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:18,007 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 215889 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1024614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:18,190 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 216052 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1118791, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:18,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 216215 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1114431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:18,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 216329 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 947204, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:18,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 216491 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1034450, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:18,831 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 78ms (threshold=20ms); ack: seqno: 216564 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 917042, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:19,004 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630652_1099518555282] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 216721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014524, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:13:19,177 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630653_1099518555283] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 216877 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 976011, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:13:19,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630653_1099518555283] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 216911 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 961756, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:13:19,321 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630653_1099518555283] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 216924 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 981319, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:13:19,496 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630653_1099518555283] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 217086 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1143462, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:13:19,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630653_1099518555283] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 217242 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1129136, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:13:19,781 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630653_1099518555283] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 217285 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2826400, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:13:19,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630653_1099518555283] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 217444 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 880518, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:13:20,522 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630653_1099518555283] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 218263 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 955914, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:13:20,585 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630653_1099518555283] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 218283 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 800035, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:13:21,138 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630654_1099518555284] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 219045 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972459, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:13:21,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630654_1099518555284] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 219155 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 895934, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:13:21,373 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630654_1099518555284] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 219165 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1125640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:13:21,555 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630654_1099518555284] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 219315 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1088798, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:13:21,737 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630654_1099518555284] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 219470 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927534, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:13:21,911 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630654_1099518555284] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 219628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1091580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:13:21,964 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630654_1099518555284] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 219652 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 682207, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:13:22,022 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630654_1099518555284] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 219659 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 846573, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:13:22,226 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630654_1099518555284] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 219814 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1083506, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:13:22,437 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630654_1099518555284] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 219974 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1034286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:13:22,610 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630654_1099518555284] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 220122 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989365, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:13:22,760 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630654_1099518555284] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 220280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946220, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:13:23,314 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630655_1099518555285] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 221027 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 676328, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:23,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630655_1099518555285] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 221300 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792010, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:24,292 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630655_1099518555285] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 222383 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 661710, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:24,686 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630655_1099518555285] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 222954 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 785725, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:25,260 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630656_1099518555286] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 223734 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 658895, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:25,796 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630656_1099518555286] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 224478 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 888060, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:25,924 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630656_1099518555286] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 224638 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 852195, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:27,097 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630657_1099518555287] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 226129 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 626603, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:27,358 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630657_1099518555287] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 226461 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 646563, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:28,317 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630658_1099518555288] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 227827 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2393783, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:28,545 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630658_1099518555288] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 228027 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1168614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:28,815 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630658_1099518555288] hdfs.DFSClient: Slow ReadProcessor read fields took 167ms (threshold=20ms); ack: seqno: 228190 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1176024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:28,984 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630658_1099518555288] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 228345 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111126, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:29,144 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630658_1099518555288] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 228468 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 891263, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:29,326 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630658_1099518555288] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 228620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043312, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:29,497 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630658_1099518555288] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 228779 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085246, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:29,669 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630658_1099518555288] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 228903 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1048479, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:29,840 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630658_1099518555288] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 229043 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827114, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:36,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630658_1099518555288] hdfs.DFSClient: Slow ReadProcessor read fields took 6555ms (threshold=20ms); ack: seqno: 229185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 648666, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:36,567 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 6521ms GC pool 'ParNew' had collection(s): count=1 time=44ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=6443ms 2015-10-16 13:13:36,778 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630659_1099518555289] hdfs.DFSClient: Slow ReadProcessor read fields took 153ms (threshold=20ms); ack: seqno: 229329 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1224053, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:37,174 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630659_1099518555289] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 229873 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 875937, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:37,313 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630659_1099518555289] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 230047 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 770190, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:37,536 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630659_1099518555289] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 230337 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 738653, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:37,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630659_1099518555289] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 230565 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 957816, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:37,929 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630659_1099518555289] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 230788 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 933885, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:38,215 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630659_1099518555289] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 231094 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 814156, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:38,377 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630659_1099518555289] hdfs.DFSClient: Slow ReadProcessor read fields took 40ms (threshold=20ms); ack: seqno: 231261 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 882283, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:38,519 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630660_1099518555290] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 231419 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 774258, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:13:38,841 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 13:13:39,149 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630660_1099518555290] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 232196 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912474, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:13:39,236 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630660_1099518555290] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 232201 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 790709, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:13:39,636 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630660_1099518555290] hdfs.DFSClient: Slow ReadProcessor read fields took 325ms (threshold=20ms); ack: seqno: 232300 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1332411, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:13:39,971 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630660_1099518555290] hdfs.DFSClient: Slow ReadProcessor read fields took 301ms (threshold=20ms); ack: seqno: 232351 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1388027, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:13:40,794 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630660_1099518555290] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 233391 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 863288, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:13:40,945 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630660_1099518555290] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 233394 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1115537, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:13:41,198 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630660_1099518555290] hdfs.DFSClient: Slow ReadProcessor read fields took 248ms (threshold=20ms); ack: seqno: 233402 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1828463, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:13:41,559 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 344ms (threshold=20ms); ack: seqno: 233414 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1316937, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:41,709 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 233576 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 842579, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:41,773 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 233606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 857388, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:41,844 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 36ms (threshold=20ms); ack: seqno: 233631 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014082, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:41,900 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 233636 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985224, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:41,939 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 233643 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 938251, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:41,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 233647 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 995640, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:42,033 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 233654 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:42,093 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 233659 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 986663, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:42,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 233672 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1057306, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:42,190 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 233679 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 883121, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:42,284 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 233696 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196015, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:42,332 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 233724 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1237926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:42,402 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 233731 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011259, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:42,545 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 233743 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1178884, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:42,624 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 233751 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 987503, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:42,665 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 233761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 918465, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:42,725 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 233765 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937837, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:42,750 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 233771 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 853192, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:42,822 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 233776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 913973, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:42,836 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: Waited 90405ms on a compaction to clean up 'too many store files'; waited long enough... proceeding with flush of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:13:42,836 DEBUG [MemStoreFlusher.1] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536927936 > lower bound=16777216 2015-10-16 13:13:42,837 INFO [MemStoreFlusher.1] regionserver.HRegion: Flushing 1/1 column families, memstore=512.05 MB 2015-10-16 13:13:42,929 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 233788 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908149, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:42,938 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 16498ms (threshold=20ms); ack: seqno: 22760 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 615023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:42,963 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 233797 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 775054, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:43,031 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 233802 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 987602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:43,057 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 233809 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911147, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:43,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 233813 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069544, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:43,239 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 233828 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 978341, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:43,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 233854 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 998900, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:43,481 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 233873 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1035779, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:43,589 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 233912 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 938309, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:43,733 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 233973 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 945318, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:44,045 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 304ms (threshold=20ms); ack: seqno: 233985 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1172578, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:44,165 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 233996 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1122273, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:44,455 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 281ms (threshold=20ms); ack: seqno: 234008 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1266375, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:44,559 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630662_1099518555292] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 1989 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 23904088, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:13:44,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630662_1099518555292] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 1999 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 19690811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:13:44,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 169ms (threshold=20ms); ack: seqno: 234021 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1227680, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:44,769 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630662_1099518555292] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 2082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 99075102, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK]] 2015-10-16 13:13:44,893 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 243ms (threshold=20ms); ack: seqno: 234033 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101459, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:44,944 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 234045 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043083, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:44,976 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 234057 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3855706, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:45,258 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#53 59081426/20543496 (287.59%), rate=51218.32 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=unlimited, activeCompactions=1] 2015-10-16 13:13:45,260 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 279ms (threshold=20ms); ack: seqno: 234070 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1395745, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:45,353 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 234081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1258157, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:45,386 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 234085 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 899482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:45,470 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630663_1099518555293] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 3249 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 699100, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:13:45,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 281ms (threshold=20ms); ack: seqno: 234094 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1577059, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:45,784 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 234105 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1255146, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:46,086 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 292ms (threshold=20ms); ack: seqno: 234118 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1011255, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:46,197 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 234130 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921474, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:46,248 INFO [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=12329560, memsize=512.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/576a7a93021040828552d0ce918816e4 2015-10-16 13:13:46,255 DEBUG [MemStoreFlusher.1] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/576a7a93021040828552d0ce918816e4 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/576a7a93021040828552d0ce918816e4 2015-10-16 13:13:46,268 INFO [MemStoreFlusher.1] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/576a7a93021040828552d0ce918816e4, entries=1864390, sequenceid=12329560, filesize=283.2 M 2015-10-16 13:13:46,269 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 3331ms (threshold=20ms); ack: seqno: 22761 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701658, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:46,270 INFO [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~512.05 MB/536927520, currentsize=0 B/0 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 3433ms, sequenceid=12329560, compaction requested=true 2015-10-16 13:13:46,270 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:13:46,270 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 12 store files, 7 compacting, 5 eligible, 10 blocking 2015-10-16 13:13:46,272 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 4 files of size 742776107 starting at candidate #1 after considering 6 permutations with 1 in ratio 2015-10-16 13:13:46,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: 127e05e2659db66008aa963b54da70f6 - family: Initiating minor compaction 2015-10-16 13:13:46,273 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegion: Starting compaction on family in region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:13:46,273 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Starting compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into tmpdir=hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp, totalSize=708.4 M 2015-10-16 13:13:46,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7d42478229094befa163faabeb602859, keycount=1864380, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=12049703 2015-10-16 13:13:46,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ea56a0f385924c7b9c4508e67d94fbcd, keycount=467760, bloomtype=ROW, size=71.1 M, encoding=NONE, seqNum=12096483 2015-10-16 13:13:46,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1b561b85de884031b2d1488825032232, keycount=466300, bloomtype=ROW, size=70.8 M, encoding=NONE, seqNum=12143117 2015-10-16 13:13:46,273 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.Compactor: Compacting hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/576a7a93021040828552d0ce918816e4, keycount=1864390, bloomtype=ROW, size=283.2 M, encoding=NONE, seqNum=12329560 2015-10-16 13:13:46,322 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2c78d320, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-10-16 13:13:46,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 98 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 879620, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:46,808 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 605ms (threshold=20ms); ack: seqno: 234142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1233277, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:46,817 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 113ms (threshold=20ms); ack: seqno: 108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908417, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:46,902 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 234153 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1076602, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:46,920 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 236 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1054664, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:46,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 234178 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 829114, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:47,289 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 290ms (threshold=20ms); ack: seqno: 363 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1377441, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:47,290 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 215ms (threshold=20ms); ack: seqno: 234306 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1464059, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:47,365 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 373 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037006, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:47,534 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 981217, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:47,535 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 169ms (threshold=20ms); ack: seqno: 234425 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1112661, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:47,623 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 234443 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014663, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:47,928 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 302ms (threshold=20ms); ack: seqno: 620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1216406, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:48,003 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 311ms (threshold=20ms); ack: seqno: 234542 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1250836, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:48,039 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 31ms (threshold=20ms); ack: seqno: 747 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 998460, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:48,131 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 765 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 716146, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:48,133 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 234586 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1232501, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:48,227 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 84ms (threshold=20ms); ack: seqno: 234597 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1022271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:48,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 779 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1074314, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:48,304 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 234608 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792414, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:48,374 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 889 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 907072, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:48,387 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 234700 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 683146, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:48,697 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 245ms (threshold=20ms); ack: seqno: 999 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1274684, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:48,715 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 322ms (threshold=20ms); ack: seqno: 234710 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1344519, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:48,770 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 234715 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037953, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:48,868 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 234721 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102604, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:48,870 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 1105 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1068747, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:48,926 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 234726 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782472, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:48,999 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 234741 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 788820, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:49,033 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 234745 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 672229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:49,087 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 234753 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 900949, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:49,288 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 264ms (threshold=20ms); ack: seqno: 1330 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1451423, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:49,289 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 200ms (threshold=20ms); ack: seqno: 234758 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 848577, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:49,348 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 234764 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 783899, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:49,437 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 234775 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 789377, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:49,476 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 234782 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 792884, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:49,532 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 234790 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930271, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:49,579 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 234824 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 806476, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:49,646 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 234831 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 830571, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:49,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 196ms (threshold=20ms); ack: seqno: 1577 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902059, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:49,674 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 234836 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762327, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:49,734 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 234842 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 772462, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:49,832 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 234854 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 800218, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:49,856 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 234859 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 744786, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:49,920 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 234866 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 928766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:50,184 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 340ms (threshold=20ms); ack: seqno: 1836 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1358340, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:50,185 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 237ms (threshold=20ms); ack: seqno: 234902 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1208792, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:50,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 42ms (threshold=20ms); ack: seqno: 1846 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 782182, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:50,311 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 234962 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1000939, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:50,349 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 1881 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 89789819, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:50,636 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 266ms (threshold=20ms); ack: seqno: 1972 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1260184, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:50,636 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 245ms (threshold=20ms); ack: seqno: 235066 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1096854, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:50,710 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 1982 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043535, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:50,807 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 85ms (threshold=20ms); ack: seqno: 1996 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 854746, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:50,808 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 235187 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 851190, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:50,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630665_1099518555295] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 2018 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1230169, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-b21f26bf-8abd-4788-a21d-9da1c49fca08,DISK]] 2015-10-16 13:13:50,978 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 235300 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1085172, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:51,211 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 216ms (threshold=20ms); ack: seqno: 2107 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1468187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:51,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 223ms (threshold=20ms); ack: seqno: 235313 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1316570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:51,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 2117 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1083731, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:51,360 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 235346 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811611, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:51,414 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 91ms (threshold=20ms); ack: seqno: 2224 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1237140, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:51,451 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 2234 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 962000, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:51,590 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 231ms (threshold=20ms); ack: seqno: 235347 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 8225541, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:51,880 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 259ms (threshold=20ms); ack: seqno: 235439 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1354324, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:51,881 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 252ms (threshold=20ms); ack: seqno: 2447 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1242452, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:51,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 235451 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861687, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:52,001 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 2556 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1113275, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:52,166 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 2668 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1098218, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:52,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630661_1099518555291] hdfs.DFSClient: Slow ReadProcessor read fields took 229ms (threshold=20ms); ack: seqno: 235490 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1037262, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:13:52,532 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 286ms (threshold=20ms); ack: seqno: 2785 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1402882, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:52,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 215ms (threshold=20ms); ack: seqno: 235646 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1159796, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:52,625 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 235663 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 976086, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:58,261 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5117ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=5598ms 2015-10-16 13:13:58,261 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 5603ms (threshold=20ms); ack: seqno: 235713 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 575945, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:58,263 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 5633ms (threshold=20ms); ack: seqno: 2907 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1487021, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:58,264 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 11994ms (threshold=20ms); ack: seqno: 22762 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1090450, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:58,383 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 2917 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1543999, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:58,393 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 235726 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1261739, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:58,434 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 2928 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 786207, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:58,470 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 235770 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 866120, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:58,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 235775 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 937097, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:58,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 195ms (threshold=20ms); ack: seqno: 235788 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 999997, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:58,739 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 98ms (threshold=20ms); ack: seqno: 3246 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1304714, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:58,840 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 235800 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1212295, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:59,080 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 212ms (threshold=20ms); ack: seqno: 235829 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1192244, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:59,081 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 116ms (threshold=20ms); ack: seqno: 3559 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1084808, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:59,180 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 235841 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 893451, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:59,235 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 3716 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 933045, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:59,402 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 167ms (threshold=20ms); ack: seqno: 3718 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1067109, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:59,404 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 175ms (threshold=20ms); ack: seqno: 235915 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993018, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:59,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 235922 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 875469, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:59,467 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 3758 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013498, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:59,611 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 136ms (threshold=20ms); ack: seqno: 3768 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1124663, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:59,612 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 104ms (threshold=20ms); ack: seqno: 236031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010441, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:59,720 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 3778 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1108279, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:59,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 236184 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:59,779 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 3799 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1110698, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:59,839 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 236240 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 908974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:13:59,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 3840 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1019732, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:59,936 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 3845 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1059633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:13:59,987 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 236390 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 879206, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:00,026 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 3907 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965654, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:00,212 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 121ms (threshold=20ms); ack: seqno: 236547 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 880135, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:00,214 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 183ms (threshold=20ms); ack: seqno: 3914 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043039, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:00,279 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 3945 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 958614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:00,367 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 96ms (threshold=20ms); ack: seqno: 236596 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 990753, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:00,369 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 3951 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1080855, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:00,400 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 236608 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 887480, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:00,437 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 3983 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 900744, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:00,498 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 3991 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972352, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:00,533 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 236767 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904353, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:00,553 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 4029 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 924435, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:00,691 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 236868 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 601780, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:00,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 26970 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 770204, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:00,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 292ms (threshold=20ms); ack: seqno: 4036 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1639500, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:00,850 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 150ms (threshold=20ms); ack: seqno: 236880 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 835666, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:00,918 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 4046 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989858, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:00,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 236973 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 969443, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:01,036 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 4076 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 984091, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:01,037 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 90ms (threshold=20ms); ack: seqno: 236982 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1150037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:01,105 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 4110 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1025702, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:01,143 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630666_1099518555296] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 4118 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 768914, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:01,461 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 308ms (threshold=20ms); ack: seqno: 237119 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1443165, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:01,462 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630668_1099518555298] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 4226 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1167595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:01,545 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 237128 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827637, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:01,709 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630668_1099518555298] hdfs.DFSClient: Slow ReadProcessor read fields took 169ms (threshold=20ms); ack: seqno: 4328 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 954459, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:01,710 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 237251 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 897324, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:01,786 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630668_1099518555298] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 4338 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 934314, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:02,101 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 311ms (threshold=20ms); ack: seqno: 237351 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1523178, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:02,105 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630668_1099518555298] hdfs.DFSClient: Slow ReadProcessor read fields took 242ms (threshold=20ms); ack: seqno: 4452 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4676703, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:02,177 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 237363 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1004063, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:02,328 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630668_1099518555298] hdfs.DFSClient: Slow ReadProcessor read fields took 146ms (threshold=20ms); ack: seqno: 4575 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1057956, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:02,347 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 92ms (threshold=20ms); ack: seqno: 237485 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 952811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:02,393 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630668_1099518555298] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 4585 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 816193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:02,469 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630667_1099518555297] hdfs.DFSClient: Slow ReadProcessor read fields took 77ms (threshold=20ms); ack: seqno: 237544 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 997878, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:02,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630668_1099518555298] hdfs.DFSClient: Slow ReadProcessor read fields took 307ms (threshold=20ms); ack: seqno: 4708 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1338210, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:02,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 231ms (threshold=20ms); ack: seqno: 237673 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1314037, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:02,813 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 237680 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930023, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:02,864 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 28969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 512681, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:02,864 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:14:02,864 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 34ms (threshold=20ms) 2015-10-16 13:14:02,866 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 237692 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 882053, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:02,941 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630668_1099518555298] hdfs.DFSClient: Slow ReadProcessor read fields took 70ms (threshold=20ms); ack: seqno: 4765 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1026075, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:02,942 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 237705 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967848, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:03,015 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630668_1099518555298] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 4775 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 972283, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:03,070 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 29030 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 660369, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:03,089 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 237793 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930111, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:03,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630668_1099518555298] hdfs.DFSClient: Slow ReadProcessor read fields took 269ms (threshold=20ms); ack: seqno: 4900 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1180518, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:03,363 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 238ms (threshold=20ms); ack: seqno: 237840 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1301335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:03,438 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630668_1099518555298] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 4910 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921341, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:03,602 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 159ms (threshold=20ms); ack: seqno: 237963 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1098859, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:03,603 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630668_1099518555298] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 5031 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 858989, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:03,679 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 237975 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1014474, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:03,999 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 239ms (threshold=20ms); ack: seqno: 238099 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1218024, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:04,000 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630668_1099518555298] hdfs.DFSClient: Slow ReadProcessor read fields took 320ms (threshold=20ms); ack: seqno: 5147 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1489806, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:04,044 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 238111 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905968, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:08,440 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4177ms GC pool 'ParNew' had collection(s): count=1 time=179ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=4206ms 2015-10-16 13:14:08,445 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630668_1099518555298] hdfs.DFSClient: Slow ReadProcessor read fields took 4391ms (threshold=20ms); ack: seqno: 5218 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1350007, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:08,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 4390ms (threshold=20ms); ack: seqno: 29872 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1248865, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:08,441 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 4387ms (threshold=20ms); ack: seqno: 238125 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 651510, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:08,589 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 133ms (threshold=20ms); ack: seqno: 238136 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1164678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:08,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 137ms (threshold=20ms); ack: seqno: 238148 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1225619, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:08,824 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 238161 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1243335, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:08,878 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 238162 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1148944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:09,079 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 195ms (threshold=20ms); ack: seqno: 238174 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1016302, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:09,182 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 238186 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 989367, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:09,218 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630668_1099518555298] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 6241 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 710766, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:09,392 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 203ms (threshold=20ms); ack: seqno: 238197 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1168791, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:09,486 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 238241 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859874, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:09,526 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 238252 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 927187, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:09,754 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 220ms (threshold=20ms); ack: seqno: 238263 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1044390, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:09,836 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 238276 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904245, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:09,940 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 99ms (threshold=20ms); ack: seqno: 238284 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 965463, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:10,138 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 191ms (threshold=20ms); ack: seqno: 238294 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 938439, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:10,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 160ms (threshold=20ms); ack: seqno: 238299 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1030511, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:10,377 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 238306 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1050426, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:10,496 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 39201 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 413678, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:10,497 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 58ms (threshold=20ms) 2015-10-16 13:14:10,497 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 58ms (threshold=20ms) 2015-10-16 13:14:10,497 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 58ms (threshold=20ms) 2015-10-16 13:14:10,497 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 58ms (threshold=20ms) 2015-10-16 13:14:10,497 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 58ms (threshold=20ms) 2015-10-16 13:14:10,498 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630670_1099518555300] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 7830 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963021, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:10,635 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 246ms (threshold=20ms); ack: seqno: 238318 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1025919, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:10,687 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 48ms (threshold=20ms); ack: seqno: 238323 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1061633, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:10,745 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 238330 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 781022, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:10,803 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 51ms (threshold=20ms); ack: seqno: 238335 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 814213, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:10,898 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 238349 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859108, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:11,058 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630670_1099518555300] hdfs.DFSClient: Slow ReadProcessor read fields took 142ms (threshold=20ms); ack: seqno: 7999 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 912740, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:11,058 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 39827 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845944, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:11,062 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 146ms (threshold=20ms); ack: seqno: 238357 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1033764, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:11,123 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 238361 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 873903, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:11,204 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 238369 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874089, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:11,314 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 238379 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 776861, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:11,415 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 238388 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932243, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:11,474 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 238395 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 820229, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:16,017 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4461ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=4538ms 2015-10-16 13:14:16,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630670_1099518555300] hdfs.DFSClient: Slow ReadProcessor read fields took 4544ms (threshold=20ms); ack: seqno: 8149 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 874673, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:16,017 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630629_1099518555259] hdfs.DFSClient: Slow ReadProcessor read fields took 4549ms (threshold=20ms); ack: seqno: 40309 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 701864, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:16,017 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 4543ms (threshold=20ms) 2015-10-16 13:14:16,017 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 4543ms (threshold=20ms) 2015-10-16 13:14:16,017 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 4543ms (threshold=20ms) 2015-10-16 13:14:16,018 INFO [sync.3] wal.FSHLog: Slow sync cost: 4543 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:16,018 INFO [sync.2] wal.FSHLog: Slow sync cost: 4543 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:16,018 INFO [sync.4] wal.FSHLog: Slow sync cost: 4543 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-168b1070-f09b-487f-a0c1-072a12b2e966,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK]] 2015-10-16 13:14:16,019 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 4546ms (threshold=20ms); ack: seqno: 238396 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1723431, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:16,084 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 238402 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1254839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:16,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 59ms (threshold=20ms); ack: seqno: 238413 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1079289, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:16,223 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630670_1099518555300] hdfs.DFSClient: Slow ReadProcessor read fields took 175ms (threshold=20ms); ack: seqno: 8178 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1285176, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:16,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 52ms (threshold=20ms); ack: seqno: 238428 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946142, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:16,311 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630670_1099518555300] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 8235 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 904310, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:16,332 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 53ms (threshold=20ms); ack: seqno: 238447 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 959825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:16,342 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:14:16,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 238458 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 864039, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:16,374 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630670_1099518555300] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 8268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 798535, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:16,388 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026341984 with entries=78024, filesize=122.31 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026456343 2015-10-16 13:14:16,390 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026326273 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026326273 2015-10-16 13:14:16,396 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026334078 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026334078 2015-10-16 13:14:16,428 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 238462 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 955420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:16,578 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630671_1099518555301] hdfs.DFSClient: Slow ReadProcessor read fields took 112ms (threshold=20ms); ack: seqno: 425 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 287008, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:16,578 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 113ms (threshold=20ms) 2015-10-16 13:14:16,578 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 114ms (threshold=20ms) 2015-10-16 13:14:16,578 INFO [sync.1] wal.FSHLog: Slow sync cost: 113 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:16,578 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 114ms (threshold=20ms) 2015-10-16 13:14:16,578 INFO [sync.2] wal.FSHLog: Slow sync cost: 113 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:16,578 INFO [sync.4] wal.FSHLog: Slow sync cost: 113 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:16,578 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 113ms (threshold=20ms) 2015-10-16 13:14:16,578 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 114ms (threshold=20ms) 2015-10-16 13:14:16,578 INFO [sync.0] wal.FSHLog: Slow sync cost: 112 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:16,578 INFO [sync.3] wal.FSHLog: Slow sync cost: 113 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:16,580 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 238477 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1103860, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:16,580 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630672_1099518555302] hdfs.DFSClient: Slow ReadProcessor read fields took 146ms (threshold=20ms); ack: seqno: 8336 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1563453, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:14:16,614 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 238483 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 821533, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:16,637 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 238484 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1115527, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:16,666 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 238495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 779873, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:16,692 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630672_1099518555302] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 8358 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 12322811, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:14:16,734 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 66ms (threshold=20ms); ack: seqno: 238500 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841631, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:16,770 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630672_1099518555302] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 8398 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839129, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:14:16,827 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 73ms (threshold=20ms); ack: seqno: 238512 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1030424, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:16,859 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630672_1099518555302] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 8437 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 826576, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:14:16,916 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 65ms (threshold=20ms); ack: seqno: 238524 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 936618, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:16,957 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630672_1099518555302] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 8480 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 773262, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:14:17,013 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630672_1099518555302] hdfs.DFSClient: Slow ReadProcessor read fields took 27ms (threshold=20ms); ack: seqno: 8523 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 729750, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:14:17,021 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 75ms (threshold=20ms); ack: seqno: 238543 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859341, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:17,122 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 238556 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 812265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:17,182 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 238562 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 963549, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:17,274 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 238568 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914956, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:17,312 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 238585 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804217, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:17,368 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 238606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915347, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:17,443 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 238628 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813482, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:17,489 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 238634 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 946982, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:17,525 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 238646 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 805728, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:17,653 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630671_1099518555301] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 5864 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 529794, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:17,653 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630672_1099518555302] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 9240 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 798614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:14:17,653 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 77ms (threshold=20ms) 2015-10-16 13:14:17,653 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 76ms (threshold=20ms) 2015-10-16 13:14:17,653 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 76ms (threshold=20ms) 2015-10-16 13:14:17,653 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 76ms (threshold=20ms) 2015-10-16 13:14:17,653 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 76ms (threshold=20ms) 2015-10-16 13:14:17,655 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 238671 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 921357, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:17,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 238682 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825646, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:17,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 238705 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 949425, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:17,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 238724 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 828946, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:18,044 DEBUG [PriorityRpcServer.handler=3,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:14:18,044 WARN [MemStoreFlusher.1] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 13:14:18,044 DEBUG [MemStoreFlusher.1] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.1; compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:14:18,208 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 238805 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 804954, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:18,308 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 238832 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841776, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:23,175 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4522ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=4691ms 2015-10-16 13:14:23,176 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 4698ms (threshold=20ms); ack: seqno: 238882 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 825287, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:23,176 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630671_1099518555301] hdfs.DFSClient: Slow ReadProcessor read fields took 4700ms (threshold=20ms); ack: seqno: 6941 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 623607, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:23,176 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 4698ms (threshold=20ms) 2015-10-16 13:14:23,176 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 4698ms (threshold=20ms) 2015-10-16 13:14:23,176 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 4698ms (threshold=20ms) 2015-10-16 13:14:23,176 INFO [sync.0] wal.FSHLog: Slow sync cost: 4697 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:23,176 INFO [sync.3] wal.FSHLog: Slow sync cost: 4697 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:23,176 INFO [sync.4] wal.FSHLog: Slow sync cost: 4697 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:23,176 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 4698ms (threshold=20ms) 2015-10-16 13:14:23,176 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 4698ms (threshold=20ms) 2015-10-16 13:14:23,176 INFO [sync.2] wal.FSHLog: Slow sync cost: 4697 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:23,176 INFO [sync.1] wal.FSHLog: Slow sync cost: 4697 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:23,181 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630672_1099518555302] hdfs.DFSClient: Slow ReadProcessor read fields took 4759ms (threshold=20ms); ack: seqno: 9535 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1397470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:14:23,264 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 238908 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008090, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:23,300 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 238921 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1053889, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:23,553 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 109ms (threshold=20ms); ack: seqno: 238992 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 811839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:23,791 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630672_1099518555302] hdfs.DFSClient: Slow ReadProcessor read fields took 140ms (threshold=20ms); ack: seqno: 10082 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 695710, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:14:23,791 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630671_1099518555301] hdfs.DFSClient: Slow ReadProcessor read fields took 139ms (threshold=20ms); ack: seqno: 9600 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 375341, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:23,791 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 139ms (threshold=20ms) 2015-10-16 13:14:23,791 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 139ms (threshold=20ms) 2015-10-16 13:14:23,791 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 139ms (threshold=20ms) 2015-10-16 13:14:23,791 INFO [sync.1] wal.FSHLog: Slow sync cost: 139 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:23,791 INFO [sync.3] wal.FSHLog: Slow sync cost: 139 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:23,792 INFO [sync.2] wal.FSHLog: Slow sync cost: 139 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:23,791 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 139ms (threshold=20ms) 2015-10-16 13:14:23,792 INFO [sync.0] wal.FSHLog: Slow sync cost: 139 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:23,793 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 227ms (threshold=20ms); ack: seqno: 239005 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 982740, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:23,935 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 132ms (threshold=20ms); ack: seqno: 239016 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 994437, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:23,948 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630671_1099518555301] hdfs.DFSClient: Slow ReadProcessor read fields took 135ms (threshold=20ms); ack: seqno: 9739 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 618759, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:24,153 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 126ms (threshold=20ms); ack: seqno: 239064 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 796797, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:24,182 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 239076 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 3337265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:24,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 239088 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 737508, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:24,396 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 239100 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101849, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:24,689 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 286ms (threshold=20ms); ack: seqno: 239111 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1223418, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:24,789 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 86ms (threshold=20ms); ack: seqno: 239124 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 923668, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:24,975 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 163ms (threshold=20ms); ack: seqno: 239136 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 867562, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:25,116 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 239148 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 861587, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:25,269 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 239173 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 867746, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:30,244 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4452ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=4632ms 2015-10-16 13:14:30,244 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630672_1099518555302] hdfs.DFSClient: Slow ReadProcessor read fields took 6207ms (threshold=20ms); ack: seqno: 10418 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 993966, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK]] 2015-10-16 13:14:30,246 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 4948ms (threshold=20ms); ack: seqno: 239185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1217286, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:30,246 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.append-pool4-t1] wal.MetricsWAL: regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.append-pool4-t1 took 4549 ms appending an edit to wal; len~=1.6 K 2015-10-16 13:14:30,248 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630671_1099518555301] hdfs.DFSClient: Slow ReadProcessor read fields took 4641ms (threshold=20ms); ack: seqno: 13678 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1149588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:30,362 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 239225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1012504, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:30,501 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 239245 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 910223, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:30,653 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 143ms (threshold=20ms); ack: seqno: 239256 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 845290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:30,890 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630671_1099518555301] hdfs.DFSClient: Slow ReadProcessor read fields took 214ms (threshold=20ms); ack: seqno: 16145 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 428440, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:30,890 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630673_1099518555303] hdfs.DFSClient: Slow ReadProcessor read fields took 214ms (threshold=20ms); ack: seqno: 10987 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 741526, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:14:30,890 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 215ms (threshold=20ms) 2015-10-16 13:14:30,890 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 215ms (threshold=20ms) 2015-10-16 13:14:30,890 INFO [sync.1] wal.FSHLog: Slow sync cost: 214 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:30,890 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 215ms (threshold=20ms) 2015-10-16 13:14:30,890 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 215ms (threshold=20ms) 2015-10-16 13:14:30,890 INFO [sync.3] wal.FSHLog: Slow sync cost: 214 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:30,890 INFO [sync.0] wal.FSHLog: Slow sync cost: 214 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:30,890 INFO [sync.2] wal.FSHLog: Slow sync cost: 214 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:30,892 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 216ms (threshold=20ms); ack: seqno: 239274 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1102136, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:30,952 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 239299 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 728605, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:31,045 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 239340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 686931, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:31,113 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 57ms (threshold=20ms); ack: seqno: 239352 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 770272, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:31,303 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 144ms (threshold=20ms); ack: seqno: 239371 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 808326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:31,442 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 239407 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 795409, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:31,724 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 239479 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 775522, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:36,434 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630671_1099518555301] hdfs.DFSClient: Slow ReadProcessor read fields took 4660ms (threshold=20ms); ack: seqno: 17891 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 587699, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:36,435 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 4659ms (threshold=20ms) 2015-10-16 13:14:36,435 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.append-pool4-t1] wal.MetricsWAL: regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.append-pool4-t1 took 4657 ms appending an edit to wal; len~=1.6 K 2015-10-16 13:14:36,435 INFO [sync.3] wal.FSHLog: Slow sync cost: 4658 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:36,435 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=1.63 MB, freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=839, evicted=0, evictedPerRun=0.0 2015-10-16 13:14:36,435 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4545ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=4654ms 2015-10-16 13:14:36,435 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 4662ms (threshold=20ms); ack: seqno: 239494 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 739561, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:36,437 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630673_1099518555303] hdfs.DFSClient: Slow ReadProcessor read fields took 4663ms (threshold=20ms); ack: seqno: 11440 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1626580, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:14:36,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 37ms (threshold=20ms); ack: seqno: 239506 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1020933, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:36,560 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630673_1099518555303] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 11492 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1000326, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:14:36,562 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 239521 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 967327, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:36,595 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#59 average throughput is 13.70 MB/sec, slept 0 time(s) and total slept time is 0 ms. 1 active compactions remaining, total limit is unlimited 2015-10-16 13:14:36,615 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 35ms (threshold=20ms); ack: seqno: 239533 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 922333, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:36,616 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/2a45d3d914da4e7a8bdc84d76f4c3fdc as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/2a45d3d914da4e7a8bdc84d76f4c3fdc 2015-10-16 13:14:36,624 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Removing store files after compaction... 2015-10-16 13:14:36,625 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Archiving compacted store files. 2015-10-16 13:14:36,627 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7d42478229094befa163faabeb602859, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/7d42478229094befa163faabeb602859 2015-10-16 13:14:36,636 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ea56a0f385924c7b9c4508e67d94fbcd, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/ea56a0f385924c7b9c4508e67d94fbcd 2015-10-16 13:14:36,644 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1b561b85de884031b2d1488825032232, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/1b561b85de884031b2d1488825032232 2015-10-16 13:14:36,653 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] backup.HFileArchiver: Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/576a7a93021040828552d0ce918816e4, to hdfs://c2020.halxg.cloudera.com:8020/hbase/archive/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/576a7a93021040828552d0ce918816e4 2015-10-16 13:14:36,653 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.HStore: Completed compaction of 4 file(s) in family of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. into 2a45d3d914da4e7a8bdc84d76f4c3fdc(size=708.3 M), total size for store is 18.3 G. This selection was in queue for 0sec, and took 50sec to execute. 2015-10-16 13:14:36,653 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6., storeName=family, fileCount=4, fileSize=708.4 M, priority=-2, time=36904335126680218; duration=50sec 2015-10-16 13:14:36,653 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: CompactSplitThread Status: compaction_queue=(0:1), split_queue=0, merge_queue=0 2015-10-16 13:14:36,653 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store files, 7 compacting, 2 eligible, 10 blocking 2015-10-16 13:14:36,654 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2015-10-16 13:14:36,654 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:14:36,654 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:14:36,660 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 239551 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 839393, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:36,699 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 239569 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 914252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:37,116 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630671_1099518555301] hdfs.DFSClient: Slow ReadProcessor read fields took 235ms (threshold=20ms); ack: seqno: 20509 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 521096, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:37,116 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 236ms (threshold=20ms) 2015-10-16 13:14:37,116 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 237ms (threshold=20ms) 2015-10-16 13:14:37,116 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 236ms (threshold=20ms) 2015-10-16 13:14:37,116 INFO [sync.3] wal.FSHLog: Slow sync cost: 235 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:37,116 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 236ms (threshold=20ms) 2015-10-16 13:14:37,116 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 236ms (threshold=20ms) 2015-10-16 13:14:37,116 INFO [sync.0] wal.FSHLog: Slow sync cost: 235 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:37,116 INFO [sync.4] wal.FSHLog: Slow sync cost: 235 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:37,116 INFO [sync.1] wal.FSHLog: Slow sync cost: 236 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:37,116 INFO [sync.2] wal.FSHLog: Slow sync cost: 236 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:37,118 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 406ms (threshold=20ms); ack: seqno: 239583 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1209068, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:37,181 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 239584 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 63852027, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:37,336 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=230675488 > lower bound=16777216 2015-10-16 13:14:37,336 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=219.98 MB 2015-10-16 13:14:37,351 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=8.00 GB, freeSize=8.00 GB, usedSize=0 B, cacheSize=0 B, accesses=1131360, hits=0, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0,cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=0, evicted=0, evictedPerRun=NaN 2015-10-16 13:14:37,379 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 195ms (threshold=20ms); ack: seqno: 239595 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1105472, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:37,425 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630671_1099518555301] hdfs.DFSClient: Slow ReadProcessor read fields took 88ms (threshold=20ms); ack: seqno: 21285 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 827209, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:37,432 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 46ms (threshold=20ms); ack: seqno: 239606 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 957279, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:37,726 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 264ms (threshold=20ms); ack: seqno: 239619 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1065804, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:37,843 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 13:14:37,843 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 13:14:37,843 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 22ms (threshold=20ms) 2015-10-16 13:14:42,845 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630669_1099518555299] hdfs.DFSClient: Slow ReadProcessor read fields took 5118ms (threshold=20ms); ack: seqno: 239620 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 852185990, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:42,845 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4729ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=4835ms 2015-10-16 13:14:42,845 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 0.8571428571428571, tune compaction throughput to 18.57 MB/sec 2015-10-16 13:14:42,847 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630671_1099518555301] hdfs.DFSClient: Slow ReadProcessor read fields took 4843ms (threshold=20ms); ack: seqno: 21973 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1134939, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:42,847 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630674_1099518555304] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 636 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1366597, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:42,936 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 239706 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 821726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:43,015 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 239727 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1053205, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:43,037 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 21ms (threshold=20ms); ack: seqno: 239730 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:43,094 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 54ms (threshold=20ms); ack: seqno: 239734 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 870614, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:43,330 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 231ms (threshold=20ms); ack: seqno: 239742 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1099961, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:43,561 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630674_1099518555304] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 1477 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 850741, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:14:43,561 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630671_1099518555301] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 25534 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 363984, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:43,561 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 100ms (threshold=20ms) 2015-10-16 13:14:43,561 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 100ms (threshold=20ms) 2015-10-16 13:14:43,561 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 101ms (threshold=20ms) 2015-10-16 13:14:43,561 INFO [sync.3] wal.FSHLog: Slow sync cost: 100 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:43,561 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 101ms (threshold=20ms) 2015-10-16 13:14:43,561 INFO [sync.0] wal.FSHLog: Slow sync cost: 100 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:43,561 INFO [sync.4] wal.FSHLog: Slow sync cost: 100 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:43,561 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 101ms (threshold=20ms) 2015-10-16 13:14:43,562 INFO [sync.2] wal.FSHLog: Slow sync cost: 100 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:43,562 INFO [sync.1] wal.FSHLog: Slow sync cost: 100 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:43,573 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 220ms (threshold=20ms); ack: seqno: 239753 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1257755, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:43,604 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 239758 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1095828, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:43,742 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 34ms (threshold=20ms); ack: seqno: 239854 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 793420, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:43,822 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.PressureAwareCompactionThroughputController: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#53 sleep 67 ms because current throughput is 21.23 MB/sec, max allowed is 18.57 MB/sec, already slept 32 time(s) and total slept time is 17707 ms till now. 2015-10-16 13:14:43,891 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 239919 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1069674, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:43,936 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=12409678, memsize=220.0 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/95f7859109c348ad815548b357234ee8 2015-10-16 13:14:43,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/95f7859109c348ad815548b357234ee8 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/95f7859109c348ad815548b357234ee8 2015-10-16 13:14:43,947 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/95f7859109c348ad815548b357234ee8, entries=801130, sequenceid=12409678, filesize=121.7 M 2015-10-16 13:14:43,948 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~220.03 MB/230718560, currentsize=37.98 MB/39829040 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 6612ms, sequenceid=12409678, compaction requested=true 2015-10-16 13:14:43,948 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:14:43,948 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 10 store files, 7 compacting, 3 eligible, 10 blocking 2015-10-16 13:14:43,949 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 1 permutations with 0 in ratio 2015-10-16 13:14:43,949 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:14:43,949 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:14:44,430 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 375ms (threshold=20ms); ack: seqno: 240086 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1406730, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:44,486 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630671_1099518555301] hdfs.DFSClient: Slow ReadProcessor read fields took 128ms (threshold=20ms); ack: seqno: 30359 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 733825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:44,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630671_1099518555301] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 32472 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 47255411, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK]] 2015-10-16 13:14:44,879 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:14:44,879 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:14:44,879 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:14:44,879 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:14:44,879 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 48ms (threshold=20ms) 2015-10-16 13:14:44,967 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 414ms (threshold=20ms); ack: seqno: 240253 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1156453, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:45,210 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 201ms (threshold=20ms); ack: seqno: 240301 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1131588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:45,470 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:14:45,503 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026456343 with entries=77900, filesize=122.11 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026485470 2015-10-16 13:14:45,504 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026341984 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026341984 2015-10-16 13:14:45,619 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#53 7143286/6829746 (104.59%), rate=6322.27 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=18.57 MB/sec, activeCompactions=1] 2015-10-16 13:14:45,621 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 406ms (threshold=20ms); ack: seqno: 240309 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1275537, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:45,837 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630676_1099518555306] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 1443 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 412683, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:45,837 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 89ms (threshold=20ms) 2015-10-16 13:14:45,837 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 89ms (threshold=20ms) 2015-10-16 13:14:45,837 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 89ms (threshold=20ms) 2015-10-16 13:14:45,837 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 89ms (threshold=20ms) 2015-10-16 13:14:45,837 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 89ms (threshold=20ms) 2015-10-16 13:14:46,043 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 417ms (threshold=20ms); ack: seqno: 240316 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1921833, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:46,275 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 177ms (threshold=20ms); ack: seqno: 240359 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 776636, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:46,276 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630676_1099518555306] hdfs.DFSClient: Slow ReadProcessor read fields took 175ms (threshold=20ms); ack: seqno: 2969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 500489, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:46,276 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 177ms (threshold=20ms) 2015-10-16 13:14:46,276 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 177ms (threshold=20ms) 2015-10-16 13:14:46,276 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 177ms (threshold=20ms) 2015-10-16 13:14:46,276 INFO [sync.4] wal.FSHLog: Slow sync cost: 176 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:46,277 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 177ms (threshold=20ms) 2015-10-16 13:14:46,277 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 176ms (threshold=20ms) 2015-10-16 13:14:46,277 INFO [sync.3] wal.FSHLog: Slow sync cost: 177 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:46,277 INFO [sync.2] wal.FSHLog: Slow sync cost: 177 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:46,277 INFO [sync.1] wal.FSHLog: Slow sync cost: 177 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:46,277 INFO [sync.0] wal.FSHLog: Slow sync cost: 176 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:47,010 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 117ms (threshold=20ms); ack: seqno: 240589 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930569, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:51,955 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 4783ms (threshold=20ms) 2015-10-16 13:14:51,955 INFO [sync.2] wal.FSHLog: Slow sync cost: 4783 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:51,956 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4609ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=4779ms 2015-10-16 13:14:51,956 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 4786ms (threshold=20ms); ack: seqno: 240652 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 762444, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:51,958 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630676_1099518555306] hdfs.DFSClient: Slow ReadProcessor read fields took 4784ms (threshold=20ms); ack: seqno: 4037 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 933349, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:52,576 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 416ms (threshold=20ms); ack: seqno: 240923 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1253347, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:52,751 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630676_1099518555306] hdfs.DFSClient: Slow ReadProcessor read fields took 792ms (threshold=20ms); ack: seqno: 4038 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 983180, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:52,751 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 794ms (threshold=20ms) 2015-10-16 13:14:52,751 INFO [sync.2] wal.FSHLog: Slow sync cost: 794 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:52,751 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 795ms (threshold=20ms) 2015-10-16 13:14:52,751 INFO [sync.1] wal.FSHLog: Slow sync cost: 794 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:52,751 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 796ms (threshold=20ms) 2015-10-16 13:14:52,751 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 795ms (threshold=20ms) 2015-10-16 13:14:52,751 INFO [sync.4] wal.FSHLog: Slow sync cost: 795 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:52,751 INFO [sync.0] wal.FSHLog: Slow sync cost: 795 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:52,751 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 793ms (threshold=20ms) 2015-10-16 13:14:52,751 INFO [sync.3] wal.FSHLog: Slow sync cost: 793 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:53,114 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 427ms (threshold=20ms); ack: seqno: 241091 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1188588, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:53,164 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 45ms (threshold=20ms); ack: seqno: 241097 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013330, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:53,259 DEBUG [PriorityRpcServer.handler=9,queue=1,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:14:53,260 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=134270176 > lower bound=16777216 2015-10-16 13:14:53,260 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=128.01 MB 2015-10-16 13:14:53,311 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630676_1099518555306] hdfs.DFSClient: Slow ReadProcessor read fields took 50ms (threshold=20ms); ack: seqno: 7076 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 810933, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:53,652 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 349ms (threshold=20ms); ack: seqno: 241258 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1101656, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:54,077 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=12456310, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/3d397da1396d4b41baf9948b819167e8 2015-10-16 13:14:54,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem: Committing store file hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/.tmp/3d397da1396d4b41baf9948b819167e8 as hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3d397da1396d4b41baf9948b819167e8 2015-10-16 13:14:54,088 INFO [MemStoreFlusher.0] regionserver.HStore: Added hdfs://c2020.halxg.cloudera.com:8020/hbase/data/default/ycsb/127e05e2659db66008aa963b54da70f6/family/3d397da1396d4b41baf9948b819167e8, entries=466280, sequenceid=12456310, filesize=70.8 M 2015-10-16 13:14:54,090 INFO [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.06 MB/134284160, currentsize=24.89 MB/26100240 for region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. in 830ms, sequenceid=12456310, compaction requested=true 2015-10-16 13:14:54,090 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:14:54,090 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 7 compacting, 4 eligible, 10 blocking 2015-10-16 13:14:54,091 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio 2015-10-16 13:14:54,091 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:14:54,091 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:14:54,190 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 395ms (threshold=20ms); ack: seqno: 241425 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1228216, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:54,684 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 62ms (threshold=20ms) 2015-10-16 13:14:54,684 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630676_1099518555306] hdfs.DFSClient: Slow ReadProcessor read fields took 61ms (threshold=20ms); ack: seqno: 14832 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 374445, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:54,684 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 62ms (threshold=20ms) 2015-10-16 13:14:54,684 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 61ms (threshold=20ms) 2015-10-16 13:14:54,684 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 61ms (threshold=20ms) 2015-10-16 13:14:54,684 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 62ms (threshold=20ms) 2015-10-16 13:14:54,728 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630675_1099518555305] hdfs.DFSClient: Slow ReadProcessor read fields took 411ms (threshold=20ms); ack: seqno: 241592 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1320382, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:54,847 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630676_1099518555306] hdfs.DFSClient: Slow ReadProcessor read fields took 125ms (threshold=20ms); ack: seqno: 15060 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 929140, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:55,069 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630676_1099518555306] hdfs.DFSClient: Slow ReadProcessor read fields took 142ms (threshold=20ms); ack: seqno: 15552 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 467148, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:55,070 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 142ms (threshold=20ms) 2015-10-16 13:14:55,070 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 142ms (threshold=20ms) 2015-10-16 13:14:55,070 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 142ms (threshold=20ms) 2015-10-16 13:14:55,070 INFO [sync.3] wal.FSHLog: Slow sync cost: 142 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:55,070 INFO [sync.2] wal.FSHLog: Slow sync cost: 142 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:55,070 INFO [sync.1] wal.FSHLog: Slow sync cost: 142 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:55,070 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 142ms (threshold=20ms) 2015-10-16 13:14:55,070 INFO [sync.4] wal.FSHLog: Slow sync cost: 142 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:14:55,266 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 388ms (threshold=20ms); ack: seqno: 241762 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1344808, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:55,804 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 402ms (threshold=20ms); ack: seqno: 241929 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1144384, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:14:56,345 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 93ms (threshold=20ms); ack: seqno: 242096 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038549, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:01,331 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4261ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=4463ms 2015-10-16 13:15:01,331 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630676_1099518555306] hdfs.DFSClient: Slow ReadProcessor read fields took 4469ms (threshold=20ms); ack: seqno: 21245 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 570471, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:15:01,331 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 4470ms (threshold=20ms); ack: seqno: 242255 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 776648, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:01,331 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 4468ms (threshold=20ms) 2015-10-16 13:15:01,331 INFO [sync.4] wal.FSHLog: Slow sync cost: 4468 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:15:01,331 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 4467ms (threshold=20ms) 2015-10-16 13:15:01,331 INFO [sync.0] wal.FSHLog: Slow sync cost: 4467 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:15:01,332 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.append-pool4-t1] wal.MetricsWAL: regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.append-pool4-t1 took 4435 ms appending an edit to wal; len~=1.6 K 2015-10-16 13:15:01,840 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 95ms (threshold=20ms) 2015-10-16 13:15:01,840 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630676_1099518555306] hdfs.DFSClient: Slow ReadProcessor read fields took 95ms (threshold=20ms); ack: seqno: 23702 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 380591, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:15:01,840 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 95ms (threshold=20ms) 2015-10-16 13:15:01,840 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 95ms (threshold=20ms) 2015-10-16 13:15:01,841 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 95ms (threshold=20ms) 2015-10-16 13:15:01,840 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 95ms (threshold=20ms) 2015-10-16 13:15:01,880 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 371ms (threshold=20ms); ack: seqno: 242430 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1052733, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:02,010 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 242514 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 915585, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:02,161 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 242570 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 881008, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:02,418 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 205ms (threshold=20ms); ack: seqno: 242597 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196958, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:02,467 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 23ms (threshold=20ms); ack: seqno: 242611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 784373, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:02,643 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 142ms (threshold=20ms); ack: seqno: 242636 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844100, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:02,856 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 191ms (threshold=20ms); ack: seqno: 242651 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1093320, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:02,881 DEBUG [PriorityRpcServer.handler=18,queue=0,port=16020] regionserver.HRegion: Flush requested on 127e05e2659db66008aa963b54da70f6 2015-10-16 13:15:02,882 WARN [MemStoreFlusher.0] regionserver.MemStoreFlusher: Region ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. has too many store files; delaying flush up to 90000ms 2015-10-16 13:15:02,882 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-10-16 13:15:02,882 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Selecting compaction from 11 store files, 7 compacting, 4 eligible, 10 blocking 2015-10-16 13:15:02,884 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio 2015-10-16 13:15:02,884 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2015-10-16 13:15:02,884 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-shortCompactions-1445024418403] regionserver.CompactSplitThread: Not compacting ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. because compaction request was cancelled 2015-10-16 13:15:02,992 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 129ms (threshold=20ms); ack: seqno: 242659 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 964036, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:03,230 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 242810 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 875507, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:03,338 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 242817 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1010274, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:03,458 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 103ms (threshold=20ms); ack: seqno: 242833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 911446, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:03,569 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 106ms (threshold=20ms); ack: seqno: 242840 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 755926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:03,684 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 110ms (threshold=20ms); ack: seqno: 242848 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 968731, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:03,897 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 100ms (threshold=20ms); ack: seqno: 242967 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 902936, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:03,976 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630676_1099518555306] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 35319 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 662531, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:15:04,068 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630676_1099518555306] hdfs.DFSClient: Slow ReadProcessor read fields took 82ms (threshold=20ms); ack: seqno: 35380 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 544704, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:15:04,068 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 243076 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 684665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:04,068 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 81ms (threshold=20ms) 2015-10-16 13:15:04,068 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 81ms (threshold=20ms) 2015-10-16 13:15:04,068 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 81ms (threshold=20ms) 2015-10-16 13:15:04,068 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 81ms (threshold=20ms) 2015-10-16 13:15:04,068 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 81ms (threshold=20ms) 2015-10-16 13:15:04,176 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 79ms (threshold=20ms); ack: seqno: 243089 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1059621, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:04,437 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 162ms (threshold=20ms) 2015-10-16 13:15:04,437 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 162ms (threshold=20ms) 2015-10-16 13:15:04,437 INFO [sync.0] wal.FSHLog: Slow sync cost: 162 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:15:04,437 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 162ms (threshold=20ms) 2015-10-16 13:15:04,437 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 162ms (threshold=20ms) 2015-10-16 13:15:04,437 INFO [sync.3] wal.FSHLog: Slow sync cost: 162 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:15:04,438 INFO [sync.4] wal.FSHLog: Slow sync cost: 162 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:15:04,437 INFO [sync.2] wal.FSHLog: Slow sync cost: 162 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:15:04,437 INFO [sync.1] wal.FSHLog: Slow sync cost: 162 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:15:04,439 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630676_1099518555306] hdfs.DFSClient: Slow ReadProcessor read fields took 164ms (threshold=20ms); ack: seqno: 36567 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 727542, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-21a30dbf-5085-464d-97f4-608a0b610c49,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-893284500-10.20.84.29-50011-1391473736889,DISK]] 2015-10-16 13:15:04,439 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 258ms (threshold=20ms); ack: seqno: 243096 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1231645, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:04,554 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 243112 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 976891, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:04,608 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:15:04,643 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026485470 with entries=77855, filesize=122.05 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026504608 2015-10-16 13:15:04,645 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Archiving hdfs://c2020.halxg.cloudera.com:8020/hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026456343 to hdfs://c2020.halxg.cloudera.com:8020/hbase/oldWALs/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026456343 2015-10-16 13:15:04,673 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 105ms (threshold=20ms); ack: seqno: 243127 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 894221, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:04,774 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 94ms (threshold=20ms); ack: seqno: 243135 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 756881, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:04,886 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 108ms (threshold=20ms); ack: seqno: 243143 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813840, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:04,989 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 97ms (threshold=20ms); ack: seqno: 243152 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859345, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:05,108 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 101ms (threshold=20ms); ack: seqno: 243160 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 932540, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:05,138 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 243162 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 763182, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:05,205 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 243174 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 841125, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:05,285 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 44ms (threshold=20ms); ack: seqno: 243185 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 813951, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:05,366 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 243196 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 850716, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:05,514 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 43ms (threshold=20ms); ack: seqno: 243221 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 875224, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:05,584 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 243233 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 702803, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:05,620 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 24ms (threshold=20ms); ack: seqno: 243239 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 750218, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:05,664 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 30ms (threshold=20ms); ack: seqno: 243245 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 807162, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:05,749 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 47ms (threshold=20ms); ack: seqno: 243257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1013470, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:05,787 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 243265 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 738051, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:05,827 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 28ms (threshold=20ms); ack: seqno: 243271 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 759444, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:05,879 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 29ms (threshold=20ms); ack: seqno: 243280 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 765276, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:10,676 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4237ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=4628ms 2015-10-16 13:15:10,759 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 4729ms (threshold=20ms); ack: seqno: 243327 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1863825, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:10,759 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630679_1099518555309] hdfs.DFSClient: Slow ReadProcessor read fields took 4714ms (threshold=20ms); ack: seqno: 2877 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1062555, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:11,396 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630679_1099518555309] hdfs.DFSClient: Slow ReadProcessor read fields took 158ms (threshold=20ms); ack: seqno: 5748 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1971207, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:11,396 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 158ms (threshold=20ms) 2015-10-16 13:15:11,396 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 159ms (threshold=20ms) 2015-10-16 13:15:11,396 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 159ms (threshold=20ms) 2015-10-16 13:15:11,396 INFO [sync.4] wal.FSHLog: Slow sync cost: 159 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:11,396 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 160ms (threshold=20ms) 2015-10-16 13:15:11,396 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 160ms (threshold=20ms) 2015-10-16 13:15:11,396 INFO [sync.1] wal.FSHLog: Slow sync cost: 159 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:11,396 INFO [sync.3] wal.FSHLog: Slow sync cost: 159 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:11,396 INFO [sync.2] wal.FSHLog: Slow sync cost: 158 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:11,396 INFO [sync.0] wal.FSHLog: Slow sync cost: 159 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:11,423 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 406ms (threshold=20ms); ack: seqno: 243602 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 856726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:11,493 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 56ms (threshold=20ms); ack: seqno: 243611 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1110885, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:11,962 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630678_1099518555308] hdfs.DFSClient: Slow ReadProcessor read fields took 345ms (threshold=20ms); ack: seqno: 243769 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043741, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:15:12,500 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 397ms (threshold=20ms); ack: seqno: 243939 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1184227, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:13,038 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 421ms (threshold=20ms); ack: seqno: 244106 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1156818, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:13,444 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630679_1099518555309] hdfs.DFSClient: Slow ReadProcessor read fields took 123ms (threshold=20ms); ack: seqno: 16697 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 752889, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:13,639 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630679_1099518555309] hdfs.DFSClient: Slow ReadProcessor read fields took 89ms (threshold=20ms); ack: seqno: 16833 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 619873, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:13,639 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 88ms (threshold=20ms) 2015-10-16 13:15:13,641 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 427ms (threshold=20ms); ack: seqno: 244273 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1168390, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:14,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 272ms (threshold=20ms); ack: seqno: 244343 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1436817, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:14,119 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630679_1099518555309] hdfs.DFSClient: Slow ReadProcessor read fields took 272ms (threshold=20ms); ack: seqno: 17072 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 802719, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:19,366 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4250ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=4723ms 2015-10-16 13:15:19,367 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.append-pool4-t1] wal.MetricsWAL: regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.append-pool4-t1 took 4611 ms appending an edit to wal; len~=1.6 K 2015-10-16 13:15:19,369 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630679_1099518555309] hdfs.DFSClient: Slow ReadProcessor read fields took 4733ms (threshold=20ms); ack: seqno: 17621 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1105220, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:19,369 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 4730ms (threshold=20ms); ack: seqno: 244520 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1572966, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:19,405 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 25ms (threshold=20ms); ack: seqno: 244527 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1166671, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:20,093 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630679_1099518555309] hdfs.DFSClient: Slow ReadProcessor read fields took 273ms (threshold=20ms); ack: seqno: 20380 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 372417, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:20,094 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 273ms (threshold=20ms) 2015-10-16 13:15:20,094 INFO [sync.3] wal.FSHLog: Slow sync cost: 272 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:20,094 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 273ms (threshold=20ms) 2015-10-16 13:15:20,094 INFO [sync.0] wal.FSHLog: Slow sync cost: 273 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:20,094 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 274ms (threshold=20ms) 2015-10-16 13:15:20,094 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 274ms (threshold=20ms) 2015-10-16 13:15:20,094 INFO [sync.2] wal.FSHLog: Slow sync cost: 273 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:20,094 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 274ms (threshold=20ms) 2015-10-16 13:15:20,094 INFO [sync.1] wal.FSHLog: Slow sync cost: 273 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:20,094 INFO [sync.4] wal.FSHLog: Slow sync cost: 273 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:20,095 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 476ms (threshold=20ms); ack: seqno: 244774 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1343174, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:20,141 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 244785 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1106784, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:20,633 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 364ms (threshold=20ms); ack: seqno: 244941 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1216146, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:20,812 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 177ms (threshold=20ms); ack: seqno: 244945 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1580027, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:20,989 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 63ms (threshold=20ms); ack: seqno: 245087 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1196839, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:21,171 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 167ms (threshold=20ms); ack: seqno: 245108 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1095587, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:21,234 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 58ms (threshold=20ms); ack: seqno: 245115 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1027846, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:21,417 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 245242 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1073468, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:21,503 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 67ms (threshold=20ms); ack: seqno: 245249 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1043224, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:21,587 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 64ms (threshold=20ms); ack: seqno: 245257 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072962, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:21,681 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 76ms (threshold=20ms); ack: seqno: 245265 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008570, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:21,775 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 74ms (threshold=20ms); ack: seqno: 245272 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1095721, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:21,876 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 72ms (threshold=20ms); ack: seqno: 245286 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 847443, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:21,995 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 80ms (threshold=20ms); ack: seqno: 245301 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1045426, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:22,093 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 81ms (threshold=20ms); ack: seqno: 245309 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1008726, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:22,185 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 71ms (threshold=20ms); ack: seqno: 245317 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 838488, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:22,398 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 196ms (threshold=20ms); ack: seqno: 245324 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 985634, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:22,398 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630679_1099518555309] hdfs.DFSClient: Slow ReadProcessor read fields took 148ms (threshold=20ms); ack: seqno: 29340 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 897182, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:22,897 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630679_1099518555309] hdfs.DFSClient: Slow ReadProcessor read fields took 279ms (threshold=20ms); ack: seqno: 29637 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758332, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:22,897 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 280ms (threshold=20ms); ack: seqno: 245404 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1461290, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:27,778 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4683ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=5145ms 2015-10-16 13:15:27,778 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.append-pool4-t1] wal.MetricsWAL: regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.append-pool4-t1 took 4873 ms appending an edit to wal; len~=1.6 K 2015-10-16 13:15:27,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630679_1099518555309] hdfs.DFSClient: Slow ReadProcessor read fields took 4876ms (threshold=20ms); ack: seqno: 29649 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 844556, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:27,780 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 4877ms (threshold=20ms); ack: seqno: 245408 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1484401, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:28,354 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 412ms (threshold=20ms); ack: seqno: 245609 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1184648, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:28,738 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630679_1099518555309] hdfs.DFSClient: Slow ReadProcessor read fields took 124ms (threshold=20ms); ack: seqno: 34499 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 738226, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-ac7407c2-80b6-43a7-8c66-3d264996d103,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-936072036-10.20.84.30-50011-1391473736999,DISK]] 2015-10-16 13:15:28,893 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 401ms (threshold=20ms); ack: seqno: 245776 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1447595, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:29,021 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630680_1099518555310] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 245878 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 684956, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-03a89da2-8ab6-465a-80bb-c83473f1dc8b,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK]] 2015-10-16 13:15:29,061 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] regionserver.LogRoller: WAL roll requested 2015-10-16 13:15:29,091 INFO [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.logRoller] wal.FSHLog: Rolled WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026504608 with entries=77832, filesize=122.01 MB; new WAL /hbase/WALs/c2021.halxg.cloudera.com,16020,1445018072095/c2021.halxg.cloudera.com%2C16020%2C1445018072095.1445026529061 2015-10-16 13:15:29,431 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 362ms (threshold=20ms); ack: seqno: 245947 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1260006, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:29,622 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 186ms (threshold=20ms); ack: seqno: 245955 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1107274, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:29,685 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 26ms (threshold=20ms); ack: seqno: 245967 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 809089, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:29,748 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 32ms (threshold=20ms); ack: seqno: 245979 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 789628, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:29,943 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 166ms (threshold=20ms); ack: seqno: 245991 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 671182, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:29,945 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630682_1099518555312] hdfs.DFSClient: Slow ReadProcessor read fields took 171ms (threshold=20ms); ack: seqno: 1837 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1509265, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:30,048 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 246011 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1038252, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:30,140 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 62ms (threshold=20ms); ack: seqno: 246023 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 925665, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:30,228 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 60ms (threshold=20ms); ack: seqno: 246035 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 905881, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:30,330 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 69ms (threshold=20ms); ack: seqno: 246048 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 909193, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:30,718 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630682_1099518555312] hdfs.DFSClient: Slow ReadProcessor read fields took 338ms (threshold=20ms); ack: seqno: 2468 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 497454, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:30,718 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 340ms (threshold=20ms) 2015-10-16 13:15:30,718 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 340ms (threshold=20ms) 2015-10-16 13:15:30,718 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 340ms (threshold=20ms) 2015-10-16 13:15:30,719 INFO [sync.0] wal.FSHLog: Slow sync cost: 339 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:30,719 INFO [sync.4] wal.FSHLog: Slow sync cost: 339 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:30,719 INFO [sync.1] wal.FSHLog: Slow sync cost: 339 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:30,721 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 362ms (threshold=20ms); ack: seqno: 246059 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1182970, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:30,803 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 55ms (threshold=20ms); ack: seqno: 246070 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 891997, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:30,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 68ms (threshold=20ms); ack: seqno: 246081 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 836464, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:35,890 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.append-pool4-t1] wal.MetricsWAL: regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.append-pool4-t1 took 4901 ms appending an edit to wal; len~=1.6 K 2015-10-16 13:15:35,891 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4674ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=4899ms 2015-10-16 13:15:35,893 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630682_1099518555312] hdfs.DFSClient: Slow ReadProcessor read fields took 4906ms (threshold=20ms); ack: seqno: 2771 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1425429, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:35,896 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 4967ms (threshold=20ms); ack: seqno: 246093 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1607632, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:36,666 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630682_1099518555312] hdfs.DFSClient: Slow ReadProcessor read fields took 313ms (threshold=20ms); ack: seqno: 5709 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 400926, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:36,666 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 314ms (threshold=20ms) 2015-10-16 13:15:36,666 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 314ms (threshold=20ms) 2015-10-16 13:15:36,666 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 314ms (threshold=20ms) 2015-10-16 13:15:36,666 INFO [sync.4] wal.FSHLog: Slow sync cost: 313 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:36,666 INFO [sync.3] wal.FSHLog: Slow sync cost: 314 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:36,666 INFO [sync.1] wal.FSHLog: Slow sync cost: 314 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:36,666 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 314ms (threshold=20ms) 2015-10-16 13:15:36,666 INFO [sync.2] wal.FSHLog: Slow sync cost: 314 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:36,667 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 315ms (threshold=20ms) 2015-10-16 13:15:36,667 INFO [sync.0] wal.FSHLog: Slow sync cost: 314 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:36,667 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 591ms (threshold=20ms); ack: seqno: 246283 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1300638, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:37,205 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 345ms (threshold=20ms); ack: seqno: 246450 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1276943, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:43,125 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4268ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=4644ms 2015-10-16 13:15:43,125 DEBUG [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] compactions.PressureAwareCompactionThroughputController: compactionPressure is 1.1428571428571428, tune compaction throughput to unlimited 2015-10-16 13:15:43,125 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 4837ms (threshold=20ms); ack: seqno: 246769 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1604828, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:43,125 WARN [c2021.halxg.cloudera.com,16020,1445018072095_ChoreService_1] regionserver.HeapMemoryManager: heapOccupancyPercent 0.98034996 is above heap occupancy alarm watermark (0.95) 2015-10-16 13:15:43,125 WARN [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.append-pool4-t1] wal.MetricsWAL: regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020.append-pool4-t1 took 4837 ms appending an edit to wal; len~=1.6 K 2015-10-16 13:15:43,129 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630682_1099518555312] hdfs.DFSClient: Slow ReadProcessor read fields took 4843ms (threshold=20ms); ack: seqno: 10279 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 916861, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:43,182 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 246775 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1225127, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:43,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 332ms (threshold=20ms); ack: seqno: 247268 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 738371, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:43,899 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630682_1099518555312] hdfs.DFSClient: Slow ReadProcessor read fields took 331ms (threshold=20ms); ack: seqno: 12893 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 524919, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:43,899 WARN [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 331ms (threshold=20ms) 2015-10-16 13:15:43,900 INFO [sync.3] wal.FSHLog: Slow sync cost: 331 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:43,900 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 331ms (threshold=20ms) 2015-10-16 13:15:43,900 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 331ms (threshold=20ms) 2015-10-16 13:15:43,900 INFO [sync.2] wal.FSHLog: Slow sync cost: 331 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:43,900 INFO [sync.0] wal.FSHLog: Slow sync cost: 331 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:43,900 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 332ms (threshold=20ms) 2015-10-16 13:15:43,900 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 332ms (threshold=20ms) 2015-10-16 13:15:43,900 INFO [sync.4] wal.FSHLog: Slow sync cost: 332 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:43,900 INFO [sync.1] wal.FSHLog: Slow sync cost: 332 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:44,659 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630682_1099518555312] hdfs.DFSClient: Slow ReadProcessor read fields took 122ms (threshold=20ms); ack: seqno: 14137 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 514905, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:50,138 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4737ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=5164ms 2015-10-16 13:15:50,140 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630682_1099518555312] hdfs.DFSClient: Slow ReadProcessor read fields took 5168ms (threshold=20ms); ack: seqno: 14452 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 603974, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:50,140 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 5171ms (threshold=20ms) 2015-10-16 13:15:50,140 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 5171ms (threshold=20ms) 2015-10-16 13:15:50,140 INFO [sync.0] wal.FSHLog: Slow sync cost: 5170 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:50,140 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 5171ms (threshold=20ms) 2015-10-16 13:15:50,140 INFO [sync.4] wal.FSHLog: Slow sync cost: 5171 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:50,140 WARN [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 5171ms (threshold=20ms) 2015-10-16 13:15:50,140 INFO [sync.1] wal.FSHLog: Slow sync cost: 5170 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:50,140 INFO [sync.2] wal.FSHLog: Slow sync cost: 5170 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:50,140 DEBUG [regionserver/c2021.halxg.cloudera.com/10.20.84.27:16020-longCompactions-1445018078835] compactions.Compactor: Compaction progress: ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6.#family#53 10169876/6829746 (148.91%), rate=7095.33 kB/sec, throughputController is DefaultCompactionThroughputController [maxThroughput=unlimited, activeCompactions=1] 2015-10-16 13:15:50,143 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630681_1099518555311] hdfs.DFSClient: Slow ReadProcessor read fields took 5173ms (threshold=20ms); ack: seqno: 247793 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1686689, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-577698b0-91fc-4c25-8d01-c9f71e45bce4,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-db741bf8-f14d-449b-91df-3bf368b79ab6,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-5a38c777-48c6-4399-a36c-a75ba628d1ca,DISK]] 2015-10-16 13:15:51,036 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630682_1099518555312] hdfs.DFSClient: Slow ReadProcessor read fields took 436ms (threshold=20ms); ack: seqno: 16674 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1384347, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:51,037 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630683_1099518555313] hdfs.DFSClient: Slow ReadProcessor read fields took 440ms (threshold=20ms); ack: seqno: 248211 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1096228, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:15:56,530 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4496ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=4570ms 2015-10-16 13:15:56,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630683_1099518555313] hdfs.DFSClient: Slow ReadProcessor read fields took 4578ms (threshold=20ms); ack: seqno: 248470 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 930855, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:15:56,530 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630682_1099518555312] hdfs.DFSClient: Slow ReadProcessor read fields took 4575ms (threshold=20ms); ack: seqno: 17632 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 517969, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:56,530 WARN [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 4575ms (threshold=20ms) 2015-10-16 13:15:56,530 WARN [sync.4] hdfs.DFSClient: Slow waitForAckedSeqno took 4575ms (threshold=20ms) 2015-10-16 13:15:56,530 INFO [sync.1] wal.FSHLog: Slow sync cost: 4575 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:56,530 WARN [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 4576ms (threshold=20ms) 2015-10-16 13:15:56,530 INFO [sync.4] wal.FSHLog: Slow sync cost: 4576 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:56,530 INFO [sync.0] wal.FSHLog: Slow sync cost: 4576 ms, current pipeline: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:56,558 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630683_1099518555313] hdfs.DFSClient: Slow ReadProcessor read fields took 22ms (threshold=20ms); ack: seqno: 248478 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1154387, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:15:57,439 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630682_1099518555312] hdfs.DFSClient: Slow ReadProcessor read fields took 451ms (threshold=20ms); ack: seqno: 19142 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1385901, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-010203bd-b42e-4fc6-b28b-0e299c1c7133,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-862206235-10.20.84.31-50011-1391473736825,DISK], DatanodeInfoWithStorage[10.20.84.30:50011,DS-0ce1972d-5ff7-4a9f-ac5a-d1233dce5137,DISK]] 2015-10-16 13:15:57,439 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630683_1099518555313] hdfs.DFSClient: Slow ReadProcessor read fields took 450ms (threshold=20ms); ack: seqno: 248757 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1072253, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:16:03,235 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4798ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=5150ms 2015-10-16 13:16:03,238 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630683_1099518555313] hdfs.DFSClient: Slow ReadProcessor read fields took 5156ms (threshold=20ms); ack: seqno: 249058 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 772164, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:16:03,282 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630683_1099518555313] hdfs.DFSClient: Slow ReadProcessor read fields took 39ms (threshold=20ms); ack: seqno: 249064 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1208167, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:16:03,998 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630683_1099518555313] hdfs.DFSClient: Slow ReadProcessor read fields took 363ms (threshold=20ms); ack: seqno: 249495 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 743508, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:16:08,906 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630683_1099518555313] hdfs.DFSClient: Slow ReadProcessor read fields took 4492ms (threshold=20ms); ack: seqno: 249943 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 678838, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:16:08,907 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4405ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=4472ms 2015-10-16 13:16:08,970 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630683_1099518555313] hdfs.DFSClient: Slow ReadProcessor read fields took 41ms (threshold=20ms); ack: seqno: 249954 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1328562, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-1676850590-10.20.84.27-50011-1391473737062,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-aa69a8eb-2761-40c7-9b18-9b887c8e5791,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d44262ac-13f5-41df-9088-263d9c679a69,DISK]] 2015-10-16 13:16:09,800 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 480ms (threshold=20ms); ack: seqno: 250337 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1558161, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:14,891 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4605ms GC pool 'ParNew' had collection(s): count=1 time=0ms GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=4965ms 2015-10-16 13:16:14,891 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 4970ms (threshold=20ms); ack: seqno: 250471 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 796876, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:20,149 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 4757ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=4960ms 2015-10-16 13:16:20,149 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 4964ms (threshold=20ms); ack: seqno: 250787 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 859596, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:22,985 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2335ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2655ms 2015-10-16 13:16:22,985 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 2660ms (threshold=20ms); ack: seqno: 250969 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 696416, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:23,034 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 33ms (threshold=20ms); ack: seqno: 250982 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1199107, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:24,629 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 1507ms (threshold=20ms); ack: seqno: 251084 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 809908, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:24,629 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1143ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1462ms 2015-10-16 13:16:26,150 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1020ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1450ms 2015-10-16 13:16:26,150 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 1454ms (threshold=20ms); ack: seqno: 251140 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 816628, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:27,506 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 1302ms (threshold=20ms); ack: seqno: 251183 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 734408, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:27,570 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 49ms (threshold=20ms); ack: seqno: 251191 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1300084, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:29,036 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1030ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1448ms 2015-10-16 13:16:29,037 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 1452ms (threshold=20ms); ack: seqno: 251208 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 758927, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:30,403 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 1350ms (threshold=20ms); ack: seqno: 251222 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 942322, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:31,869 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 1464ms (threshold=20ms); ack: seqno: 251225 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1742013, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:33,375 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher: Waited 90494ms on a compaction to clean up 'too many store files'; waited long enough... proceeding with flush of ycsb,,1445024377274.127e05e2659db66008aa963b54da70f6. 2015-10-16 13:16:33,375 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1008ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1500ms 2015-10-16 13:16:33,375 DEBUG [MemStoreFlusher.0] regionserver.FlushLargeStoresPolicy: Flush Column Family family of 127e05e2659db66008aa963b54da70f6 because memstoreSize=536904576 > lower bound=16777216 2015-10-16 13:16:33,375 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 1504ms (threshold=20ms); ack: seqno: 251228 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1104973, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:33,376 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=512.03 MB 2015-10-16 13:16:35,105 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1011ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1719ms 2015-10-16 13:16:35,105 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 1723ms (threshold=20ms); ack: seqno: 251233 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1177788, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:36,460 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 1351ms (threshold=20ms); ack: seqno: 251237 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2123442, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:37,901 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 1441ms (threshold=20ms); ack: seqno: 251238 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1938458, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:16:39,249 WARN [ResponseProcessor for block BP-410607956-10.20.84.26-1391491814882:blk_1080630684_1099518555314] hdfs.DFSClient: Slow ReadProcessor read fields took 1348ms (threshold=20ms); ack: seqno: 251239 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2108112, targets: [DatanodeInfoWithStorage[10.20.84.27:50011,DS-01b777e4-2a05-4192-bbf2-1eadb054ca92,DISK], DatanodeInfoWithStorage[10.20.84.31:50011,DS-dc00cfc0-9479-4786-ac3b-0571013c4e38,DISK], DatanodeInfoWithStorage[10.20.84.29:50011,DS-d9a6905d-c676-46bd-a791-7de2b27973f5,DISK]] 2015-10-16 13:17:01,275 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2015-10-16 13:17:01,275 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=c2021.halxg.cloudera.com 2015-10-16 13:17:01,275 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_60 2015-10-16 13:17:01,275 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2015-10-16 13:17:01,275 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/home/stack/bin/jdk1.8.0_60/jre 2015-10-16 13:17:01,275 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/home/stack/hbase/bin/../conf:/home/stack/bin/jdk//lib/tools.jar:/home/stack/hbase/bin/..:/home/stack/hbase/bin/../lib/activation-1.1.jar:/home/stack/hbase/bin/../lib/aopalliance-1.0.jar:/home/stack/hbase/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/home/stack/hbase/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/stack/hbase/bin/../lib/api-asn1-api-1.0.0-M20.jar:/home/stack/hbase/bin/../lib/api-util-1.0.0-M20.jar:/home/stack/hbase/bin/../lib/asm-3.1.jar:/home/stack/hbase/bin/../lib/avro-1.7.4.jar:/home/stack/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/home/stack/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/stack/hbase/bin/../lib/commons-cli-1.2.jar:/home/stack/hbase/bin/../lib/commons-codec-1.9.jar:/home/stack/hbase/bin/../lib/commons-collections-3.2.1.jar:/home/stack/hbase/bin/../lib/commons-compress-1.4.1.jar:/home/stack/hbase/bin/../lib/commons-configuration-1.6.jar:/home/stack/hbase/bin/../lib/commons-daemon-1.0.13.jar:/home/stack/hbase/bin/../lib/commons-digester-1.8.jar:/home/stack/hbase/bin/../lib/commons-el-1.0.jar:/home/stack/hbase/bin/../lib/commons-httpclient-3.1.jar:/home/stack/hbase/bin/../lib/commons-io-2.4.jar:/home/stack/hbase/bin/../lib/commons-lang-2.6.jar:/home/stack/hbase/bin/../lib/commons-logging-1.2.jar:/home/stack/hbase/bin/../lib/commons-math-2.2.jar:/home/stack/hbase/bin/../lib/commons-math3-3.1.1.jar:/home/stack/hbase/bin/../lib/commons-net-3.1.jar:/home/stack/hbase/bin/../lib/curator-client-2.7.1.jar:/home/stack/hbase/bin/../lib/curator-framework-2.7.1.jar:/home/stack/hbase/bin/../lib/curator-recipes-2.7.1.jar:/home/stack/hbase/bin/../lib/disruptor-3.3.0.jar:/home/stack/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/home/stack/hbase/bin/../lib/gson-2.2.4.jar:/home/stack/hbase/bin/../lib/guava-12.0.1.jar:/home/stack/hbase/bin/../lib/guice-3.0.jar:/home/stack/hbase/bin/../lib/guice-servlet-3.0.jar:/home/stack/hbase/bin/../lib/hadoop-annotations-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-auth-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-client-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-hdfs-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-app-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-core-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-jobclient-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-shuffle-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-api-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-client-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-server-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hbase-annotations-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-annotations-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-client-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-common-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-common-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-examples-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-external-blockcache-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-hadoop2-compat-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-hadoop-compat-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-it-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-it-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-prefix-tree-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-procedure-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-protocol-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-resource-bundle-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-rest-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-server-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-server-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-shell-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-thrift-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/htrace-core-3.1.0-incubating.jar:/home/stack/hbase/bin/../lib/httpclient-4.2.5.jar:/home/stack/hbase/bin/../lib/httpcore-4.2.4.jar:/home/stack/hbase/bin/../lib/jackson-core-asl-1.9.13.jar:/home/stack/hbase/bin/../lib/jackson-jaxrs-1.9.13.jar:/home/stack/hbase/bin/../lib/jackson-mapper-asl-1.9.13.jar:/home/stack/hbase/bin/../lib/jackson-xc-1.9.13.jar:/home/stack/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/home/stack/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/home/stack/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/home/stack/hbase/bin/../lib/javax.inject-1.jar:/home/stack/hbase/bin/../lib/java-xmlbuilder-0.4.jar:/home/stack/hbase/bin/../lib/jaxb-api-2.2.2.jar:/home/stack/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/stack/hbase/bin/../lib/jcodings-1.0.8.jar:/home/stack/hbase/bin/../lib/jersey-client-1.9.jar:/home/stack/hbase/bin/../lib/jersey-core-1.9.jar:/home/stack/hbase/bin/../lib/jersey-guice-1.9.jar:/home/stack/hbase/bin/../lib/jersey-json-1.9.jar:/home/stack/hbase/bin/../lib/jersey-server-1.9.jar:/home/stack/hbase/bin/../lib/jets3t-0.9.0.jar:/home/stack/hbase/bin/../lib/jettison-1.3.3.jar:/home/stack/hbase/bin/../lib/jetty-6.1.26.jar:/home/stack/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/home/stack/hbase/bin/../lib/jetty-util-6.1.26.jar:/home/stack/hbase/bin/../lib/joni-2.1.2.jar:/home/stack/hbase/bin/../lib/jruby-complete-1.6.8.jar:/home/stack/hbase/bin/../lib/jsch-0.1.42.jar:/home/stack/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/home/stack/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/stack/hbase/bin/../lib/jsr305-1.3.9.jar:/home/stack/hbase/bin/../lib/junit-4.12.jar:/home/stack/hbase/bin/../lib/leveldbjni-all-1.8.jar:/home/stack/hbase/bin/../lib/libthrift-0.9.2.jar:/home/stack/hbase/bin/../lib/log4j-1.2.17.jar:/home/stack/hbase/bin/../lib/metrics-core-2.2.0.jar:/home/stack/hbase/bin/../lib/netty-all-4.0.30.Final.jar:/home/stack/hbase/bin/../lib/paranamer-2.3.jar:/home/stack/hbase/bin/../lib/protobuf-java-2.5.0.jar:/home/stack/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/stack/hbase/bin/../lib/servlet-api-2.5.jar:/home/stack/hbase/bin/../lib/slf4j-api-1.7.7.jar:/home/stack/hbase/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/stack/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/home/stack/hbase/bin/../lib/spymemcached-2.11.6.jar:/home/stack/hbase/bin/../lib/xercesImpl-2.9.1.jar:/home/stack/hbase/bin/../lib/xml-apis-1.3.04.jar:/home/stack/hbase/bin/../lib/xmlenc-0.52.jar:/home/stack/hbase/bin/../lib/xz-1.0.jar:/home/stack/hbase/bin/../lib/zookeeper-3.4.6.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/etc/hadoop:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/servlet-api-2.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/log4j-1.2.17.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-codec-1.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jettison-1.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jersey-server-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/curator-client-2.6.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/activation-1.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jersey-json-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-lang-2.6.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jetty-6.1.26.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-net-3.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/junit-4.11.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/paranamer-2.3.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-digester-1.8.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/htrace-core-3.0.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/hadoop-annotations-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jsch-0.1.42.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/gson-2.2.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/curator-framework-2.6.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/hadoop-auth-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jsp-api-2.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/avro-1.7.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jersey-core-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-io-2.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/xz-1.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/commons-cli-1.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/xmlenc-0.52.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/asm-3.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/guava-11.0.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/hadoop-nfs-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/hadoop-common-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/common/hadoop-common-2.7.0-SNAPSHOT-tests.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/asm-3.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/hadoop-hdfs-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/hdfs/hadoop-hdfs-2.7.0-SNAPSHOT-tests.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jettison-1.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/guice-3.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/activation-1.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/javax.inject-1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jline-0.9.94.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/xz-1.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/asm-3.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-common-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-registry-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-client-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-api-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-common-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.0-SNAPSHOT-tests.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.0-SNAPSHOT.jar:/home/stack/hadoop-2.7.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.0-SNAPSHOT.jar:/home/stack/hadoop/contrib/capacity-scheduler/*.jar 2015-10-16 13:17:01,276 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/home/stack/hadoop-2.7.0-SNAPSHOT/lib/native 2015-10-16 13:17:01,276 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2015-10-16 13:17:01,276 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler= 2015-10-16 13:17:01,276 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2015-10-16 13:17:01,276 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2015-10-16 13:17:01,276 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-358.23.2.el6.x86_64 2015-10-16 13:17:01,276 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=stack 2015-10-16 13:17:01,276 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/stack 2015-10-16 13:17:01,277 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/stack/hbase-2.0.0-SNAPSHOT 2015-10-16 13:17:01,278 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=c2020.halxg.cloudera.com:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@2cbb3d47 2015-10-16 13:17:01,373 INFO [main-SendThread(c2020.halxg.cloudera.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server c2020.halxg.cloudera.com/10.20.84.26:2181. Will not attempt to authenticate using SASL (unknown error) 2015-10-16 13:17:01,388 INFO [main-SendThread(c2020.halxg.cloudera.com:2181)] zookeeper.ClientCnxn: Socket connection established to c2020.halxg.cloudera.com/10.20.84.26:2181, initiating session 2015-10-16 13:17:01,400 INFO [main-SendThread(c2020.halxg.cloudera.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server c2020.halxg.cloudera.com/10.20.84.26:2181, sessionid = 0x15071c9e67400bc, negotiated timeout = 30000